Amazing returns = superior stock selection strategy + superior in & out strategy

Good discussion here Guy Fleury and Peter Guenther. While also not saying that this strategy is good or bad, Guy Fleury makes an irrefutable point that what matters is the actual, or out-of-sample performance, and we can agree that 2022 was beyond awful for the strategy as it existed in Nov 2021. 

At least part of this is because, ironically, a longer backtest does not make a better backtest. And, CAGR over the entire period is actually a pretty poor measure of performance. It is much better to test a strategy (this goes for any model based on historical data, not just trading models) using training sets, validation sets and hold-out sets looking at many different such stratification selections, and then using a performance metric that incorporates both the return and drawdown in its calculation (or, for example, a metric that maximizes CAGR while minimizing the year-to-year CAGR variance). As it was, the 2008-2021 backtest ranked by CAGR couldn't help but overfit: much better would have been to look at several 3-year periods within that timeframe, tuning the model for its performance on other (not necessarily immediately subsequent) periods, looking for a set of results that achieved robust performance regardless of the validation time period.

In addition to the testing methodology described above, I keep coming back to what I see as a more fundamental question of algorithm design… the simply stated (though sometimes difficult to achieve) exercise of looking for holes or “what will kill the thing?” In this case, as I said in earlier comments, I believe that the twin assumptions of 1) continuing negative correlation between equity returns and bond returns; and 2) the continuing assumption of low inflation were the likely culprits. 

Now that we have a stress scenario (this year) with those to assumptions blown, it would now be useful to choose a timeframe, like Jan 2019 - Sept 2022 as our training set, and then go back and test the model on segments of the 2008-2020 timeframe. Might be that we get decent results, might be we don't; I haven't tried it yet. And then we can ask ‘what will now kill this thing?’ and see what we might come up with. My standard go-to in this situation is stagflation ala Japan of the last 30 years… if that happened, how might we expect the model to respond?

That sounds sensible, Mark Kust. Are you working on these kinds of tests by any chance at the moment, and would you be happy to share results in this thread? That would be very interesting material to look at.

As mentioned before, I propsed the following two changes:

• do not trade TMF or TLT if there is stat. significant yield inversion
• do not trade TMF or TLT if inflation is over 6%

it avoids most of the 2022 drawdown and seems not to impact performance 2008-2022. However, I forgot to do a performance analysis on 2002-2010. 

Regarding performance metric I am still struggling a lot to find a good solution. To compare performance of regimes/samples I currently use

• 95% confidence intervall of 1-y rolling sortino ratio (in daily steps)
• 95%  confidence intervall of 1-y rolling (compounding annual return % / max-drawdow.totalperiod)

Can you please share the algo with your changes Santa24 ?

I think that people have different argumentation angles here. A quick metaphor might help to disentangle. 
Let’s use a car: 
> Guy Fleury asserts that the car has crashed and concludes that the engine (the signals) is faulty and an airbag is required.
> I note that the engine was working fine and that the car’s features are the engine, a gas pedal (in selection), and a brake pedal (out selection). I argue and show that it was the brake pedal, not the engine, that was not working properly.
> Santa24 notes that the brake pedal can be fixed by a certain means.
> Mark Kust argues that the car needs to be tested more, e.g. on sub-sections of the road or completely different roads but maybe even on ice or water. He proposes that the car is only fully reliable if it can master all these different conditions.

I just need to drive the latest car  😊 

Yes, the engine failed, the car crashed (it was totaled) and there were no airbags. But airbags might not have been sufficient. Already, the composite signal of the In & Out strategy had for primary objective to protect from drawdowns. It was even the strategy's mantra: signal goes down, you get out of stocks; signal goes up, you move back in. For when the market was not moving up, you were in bonds or cash equivalents with low return expectations. Over last year, the signal (engine or whatever) was wrong 86% of the time. That makes the signal composition assumptions faulty and in need of repair, redesign, or at least reassessment.

I currently see the In & Out strategy as a classic case of curve fitting to the extreme. It had many modifications and tweaks while being developed on the Quantopian website. Its trading signals were fine-tuned over what should be considered its training and validation periods prior to October 2020. Based on portfolio metrics generated by those simulations prior to October 2020, it seemed to work fine, showed promise, including CAGR-wise.

This particular version was ported to QuantConnect in October 2020 and also had quite a few modifications. We could consider that from October 2020 to October 2021 (a whole year), the strategy was further refined and had its validation and walk-forward phase (a year is more than enough for that). Should we design these things to put them on walk-forward mode for years on end? At one point, those strategies have to go live or else be discarded.

In my previous post, I did not want to be too harsh which is why I used the term "walk forward" when describing the In & Out strategy. What I intended to use was "going live" after the strategy's one-year "walk forward" leading to October 2021. The strategy has been frozen in time since October 2021 and therefore any simulation up to October 2022 would be like if having traded this strategy live with its consequences (good or bad).

A simple simulation from October 2021 to October 2022 was sufficient to show that the strategy was clearly overfitted. It is the most reasonable explanation. From what should have been its live phase, the strategy went from an 86% win rate prior to October 2021 to a -86% loss rate from October 2021 to October 2022. This means that 86% of the trading signals were wrong or mistimed. Those were not sought-after portfolio metrics, especially if trading live.

Whatever this strategy was or has been up to October 2021, it does not matter so much for the person that traded that thing live from October 2021 to October 2022. It was simply a disaster, a real train wreck, a portfolio destroyer, providing no reprise.

What can be done now is a re-design, an overall restructuring. We could use the information that has been gained over those 14 years. But first, there is a need to demonstrate that the composite trading decision signal has value, not of the coincidental type once in a while but from a statistically significant point of view. However, there might not be sufficient data to make that case.

Here is the equation for this strategy, or any other, for that matter:

Guy Fleury the engine did not fail, unless this engine is severely different from the in / out thread, i believe the signals are the same. Algo versions I've posted in that thread only have a -11% drawdown for this whole insane year. Obviously not every signal is going to be a winner no idea where you are getting -80% drawdowns from.Nor that it matters, because what matters is money management and drawdowns, those algos in those threads are doing about 20% returns 20% drawdowns from 08 - now

@Jack, my comments have been about this particular version of the In & Out strategy.

I have not retested the other 4 versions to see their outcomes over the last year. They are basically different strategies that should give different results. Maybe some of their protective routines might help this version of the program. I do not know. But there, simulations and validation procedures will have to be done... to get an idea of what to expect going forward.

Your comment does raise a more interesting question: which strategies should one choose out of the 5 In & Out strategies or out of the tens of thousands out there? They will all give different results but you might be interested in only the few top performers going forward. We are still playing for the highest profit possible even with no guarantees.

If you find my presented data inaccurate, you can just redo the given simulation based on the October 2021 version which you will find here. You are expected, evidently, to get the same answers I got.

Here is also the URL for the results based on this version of In & Out over the last year only.

From its portfolio metrics, I cannot say it was a successful strategy with its -81% drawdown, its 86% loss rate, or its -1.27 Sharpe ratio. I do not think I would be alone to say this.

Definitely, something did not work out as planned... or even as expected...

Now, how can this strategy be repaired? Or is it even repairable?

The algorithm relied on bonds (TLT) which had an absolutely terrible run since YTD if you look at the charts.  Normally TLT is a safe haven during a recession, but not this time. The algorithm is still certainly still savlagable.  Instead of just SetHoldings to TLT, you could just do some balancing into a “defensive” bucket of stocks, similar to what a momentum algorithm does in the place of the self.HLD_OUT logic

I could see this getting combined with how this algorithm does a momentum strategy when picking which holding to take.  One could swap that out for  the Hold Out weight logic 
 

Glenn Vinson I am trying to run your algo live but keep getting into this error

Runtime Error: 'QLD' wasn't found in the Slice object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("QLD") in Slice.cs:line 315 Stack Trace: 'QLD' wasn't found in the Slice object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("QLD") in Slice.cs:line 315

Any clues?

Updated reflections
It's been a while since I shared the algo's live trading performance in this post here. Some further reflections: Good news is that I think one can make money based on the algo. Bad news is that one needs to actively manage and intervene. Bummer. Current approach: I use the algo's In/Out indicator as a “proposed course of action”, but then consider current news (eg. currently about inflation, the economy etc.) to make a final decision. Sometimes I decide to “drag my feet” with regard to going in/out, eg based on the current charts of the S&P-500 and/or Nasdaq-100 indices. Also, I am avoiding bonds at the moment. Of course, this makes it less clean and more difficult to share the strategy. One additional thought: I am using Fed interest rate increases as an additional out signal (two trading weeks out = 10 days). Good luck!
 

Peter Guenther thanks for the update,  are you using V8 version of this algo?

You are right Rulak. For the ‘proposed’ in/out moves, it's the v8 plus interest rate hikes by the Fed, and bonds set to a zero weight. But you will have to assess these proposed moves critically and make a call / sometimes overrule. Chart analysis can be useful, and it might be possible to integrate this directly into the algo, but this would require input from an expert in that area.

Peter Guenther would you mind to post the mention algo (V8) as a reverence as I’m loosing a bit to overview - thanks

No worries, Carsten, I think it should be this one here. Then along the lines discussed above: add recent rate hikes and use as a ‘proposed course of action’ that you challenge critically, eg based on current news and chart analysis.

Love the thinking and work that's gone into this strategy. Thanks all. 

IMHO this recent performance issue is simply because the strategy has never been running or backtested in an inflationary environment with rising interest rates. The timing switches between stocks (QQQ - nasdaq/tech sector index) or bonds (TLT or the shorter maturity bond index) and the premise here is that stocks and bonds have inverse returns correlation. That broke down in 2022 (as it does periodically, generally in rising interest rate / inflationary market regimes). I think the timing mechanism is sound, you just need another instrument to trade instead of bonds when the stock bond correlation is positive. 

Basically - use In/out timing. if timing says out of the stock market check the stocks bonds correlation - if negative - trade - bonds, if positive - trade commodities or some other market.

Simon David 

The issue is bigger than the choice of “out” assets, the in/out signal itself has been misfiring since the start of 2022 with sustained equity losses when “in”.  You can see that clearly in the attached backtest, starting from the same workbook that Peter linked to but going to cash (BIL) on out signals.

Yep, that's exactly what I'm saying. Equities (especially growth equities, e.g. QQQ index) have been losing since mid 2022 because interest rates are rising and this significantly impacts the expected future earnings of growth (e.g. technology) companies. Bonds also, as they lose coupon value as interest rates rise. So both the risk on and safe haven instruments are now correlated (albeit for different reasons). Both the “in” and “out” instruments are losing under the current market regime, regardless of the signal. The “in” signal is suggesting equities are going to have better returns based on economic factors, and these factors still hold, so you get an “in” (or risk on) signal. The signal itself isn't terrible, but the instrument chosen to trade it is being affected by other factors (i.e. the market is rotating out of growth equities, into energy stocks or consumer durables etc). I'm suggesting that the strategy could be extended in (growth equities)/out (bonds) during non inflationary regimes, hold something else (gold?) or short bonds during rising interest rate regimes.

Tried building on this but focusing on an algorithim that will pick a list of stocks based off of fundamental fine filter criteria to then get passed to a list of technical indicators as opposed to specific etfs, however its breaking somewhere along the ways of being populated after the fine filter.  See attached:

The main idea was to no longer let the self.rebalance function setHoldings, but rather have that passed to a seperate function “buySignals” to check if it passes technical indicators to enter trade.  self.symbols list of symbols doesn't seem to be getting passed through the initialize block.

Line 249 debug shows that the coarse filter is definitely populating stocks, so there seems to be some disconnect.