Amazing returns = superior stock selection strategy + superior in & out strategy

Hi Peter, thank you very much. This is very helpful. I have backtesting now. You have all done great work here. thank you again.

Hi @xeno-core , could you share the working clone of qualup_inout_v5.  Thanks!

Hi WC, 

Sure, here it is….they have done some interesting work here. 

A Walk-forward Example
I would like to use a version of the "In & Out" strategy (frozen in time since October 2021) as a one-year walk-forward example.
The price data after October 2021 will be totally unknown to the strategy. It is assumed the strategy will continue to do what it was designed for and simply execute its code. After all, it was designed to do just that.

The strategy received a lot of praise for its outstanding performance: 83.198% CAGR over the period from 2008 to October 2021, and this, with an 86% win rate. Exceptional. Just based on this remarkable CAGR performance and portfolio metrics, people might have started trading it live at the time. Hopefully not.

Payoff Matrix Equation

The above payoff matrix equation covers the period up to τ (October 2021) to which is then added its walk-forward part up to t (meaning current date).

The first part of the payoff matrix covers the first n trades up to time τ. The period after τ has all the other trades (N - n) that occurred after October 2021). N is the total number of trades over the entire trading interval up to t.

The overall expectation was that the market, going forward, would behave as it did over its past 13 years. And therefore, the trading procedures would continue to do what they did, and this is: continue to prosper.

We should expect the market average rm to stay about the same over the period. The same goes for the strategy's alpha (its excess return). Nonetheless, the market average during the walk forward could still be different. It is only if (t - τ) would be large enough (some 20+ years) that we could expect rm to tend to its long-term average expectation. Over the short term (one year), it might not be the case, as this case shows.

Based on the above equation, you might not be able to evaluate the walk-forward period without actually living it. Any estimation for the period (t - τ) might be just an extrapolation,
some kind of wishful forward disposition with absolutely no guarantees that it could even happen.

You have to live that period (t - τ) doing live trades just to see what would have happened. And if you did, there would be only one solution, and that is the above equation where all the variables would be identified with their before and after values. The only outcome would be this walk forward since trading strategies by their very structure (programs) only consider one portfolio path.

To demonstrate the equation, I used the "In & Out" strategy (October 2021 version). Here is its QuantConnect URL address and code: Amazing returns Strategy as of October 2021.

The cited simulation shows its results up to October 2021, and below you will have what happened next. The strategy did not know and could not know what was coming its way. It would have to make do following its preset trading procedures. 

We can observe a total breakdown of the strategy with an -85% drawdown. By definition, no one is intended to trade during a walk forward, it is just another phase in the simulation and testing process. But, for those that opted to trade that strategy live without first adding downside protection, it must have been misery. 

The strategy still made money, but it was down -85% from its peak after October 2021. That was somewhere around a $455 million loss for a strategy that started on a $100,000 initial stake. If you added some 30% leverage as could be used in a 130/30 strategy, the drawdown would have been -92%. Almost a total collapse.

So, What Went Wrong?

The initial assumptions made, the trading rules, and what was classified as trade-triggering signals did not hold up going forward. Yet, the trading environment from 2008 to October 2021 had a generally upmarket for most of the period.

It was as if most of the trading signals were wrong, as I already stated in another thread on another version of the "In & Out" strategy. But, in an upmarket, being wrong did not matter so much. The tide can lift all boats as some say.

I think what was mostly missing were protective measures for when the market might be heading south or when the logic behind the signals would stop working. We should design trading strategies for what is coming next and not necessarily for what was. A simulation is just to show how your strategy would have behaved, showing you what might have happened if such and such. The important part is what happens after.

Another point of interest would be that the code modifications made after October 2021 were to improve the program based on the knowledge of the added data points. And that, in reality, becomes incremental curve fitting, but curve fitting nonetheless. The "oh, let me arrange that" after the fact is not a viable scenario in trading.

This is not an attack on this trading strategy. What it is, however, is an observation, an example of what happened before τ and what followed. It provides a perfect example of a walk forward.

There are at least 5 versions of the "In & Out" strategy. In one of them, last year, I recommended adding downside protection and further improving trading signals before use. I estimated at the time that 80% of the trading signals were wrongly timed. It would appear it was not bad advice.

Should you add downside protection to your trading strategies? Definitely.

Thanks for sharing, Guy Fleury. Downward protection is definitely one for the algo’s to-do list. Walk-forward analyses are highly recommendable. One aspect that caught my eye: You mention that the signals were wrong and that 80% of them were wrongly timed. If you are happy to share, it would be interesting to learn more about what analyses you ran that indicated these issues. It is an interesting one for me as it is the opposite of my observations related to this current downturn. In an earlier post, I have written about the algo’s three distinct components, noting that the signals are used for the algo’s market timer, not for the in holdings selection nor for the out holdings selection. In my personal trading, I use the market timer and this component has done a relatively precise job, as noted in another post. In my view, the one component that got the algo into trouble in this recent downturn is the out holdings selection, which is not informed by any signals but instead simply hard-wired to TLT. It seems that too little, instead of too much, (curve) fitting has gone into this component, no?

Peter Guenther to properly gauge the hypothesis i would try and go back 100 years of data on the market timer.

Absolutely, Jack Pizza, such an analysis would be amazing. Are you working on something along those lines by any chance?
One key challenge is that many of the ETFs used to generate the signals began trading around 2008. So, you would need to construct something like a “synthetic ETF”, i.e. a portfolio that follows the same portfolio construction logic as the ETF that you want to mimic. With some dedicated work it can be done. However, beware of survival bias. If the ETF is based on stocks, you will need historic stock price data for all the firms that existed at a certain point in time, not only those that survived until today.
Tip: You could check out the main In & Out thread. I recall that there has been a discussion about testing the algo for earlier time periods and an initial dataset was shared that you might be able to build on. As an initial test you might not even need to go back 100 year. The 80s and 70s had a fair amount of (equity market) crises, so a test on these time periods would already be quite insightful. Imagine the market timer would work fine for these time periods as well … we should consider starting our own mutual fund then as this would be simply amazing 🚀

@Peter, from my post the question was: What Went Wrong? We could say that the premises on which the strategy was built did not hold or did very badly going forward, or, at a minimum wrongly postulated or aggregated. 

The version I analyzed did rather well over its first 13 years. It survived the 2008 financial crisis, handled a number of minor dips, and even the more serious Covid-19 dip.

From October 2021, looking at the strategy, as it was then, one would only have had a positive disposition looking in awe at the strategy's 83.198% CAGR. An outcome that greatly surpassed even Rentec's Medallion Fund 64% CAGR (39% net after expenses).

The provided payoff matrix equation stated that even if you sliced your trading strategy, the trading behavior should have been about the same: F(14) = 100,000 ∙ (1 + 0.83)^13 ∙ (1 + 0.83)^1. The strategy was hard coded and therefore should have generated about the same number of trades per period with about the same variance and therefore about the same profits. 

However, what we got was: F(14) = 100,000 ∙ (1 + 0.83)^13 ∙ (1 – 0.85)^1. What amounts to a total reversal of its 13-year trend. Reducing the portfolio by 85% in a single year. We have to put forward that the trading signals used totally broke down.

And from there, we have to assume that the premises on which the signals were based not only stopped working going forward, but also reversed. No one corrected them or realized that they were not working anymore. It is only with the provided walk-forward chart that we started questioning the trading procedures.

However, another question should have been asked: Is a trade triggering signal wrong when you could generate higher overall profits if it was absent? Or, even if wrongly timed, still generate more profits?

If I remember correctly (lost a few of those backtests last spring for some reason), I got rid of most of what composed the trade-triggering mechanics of the version I used. Only kept -UUP as signal generator. It was mostly going down during the whole trading interval (the reason for using the minus sign). I was looking for an upward straight line at the time to replace the signals, but did not know how to integrate it as the strategy's guideline. So I settled on using -UUP since it had low variance and, on average, pointed upward. This also meant I looked at the data over the trading interval and, in so doing, introduce a forward bias knowing that -UUP would be going up beforehand.

Using -UUP improved overall performance compared to using the sum of the other signals. But this also said that at one point in its future, UUP would stop going down and create its own set of problems. It was the main reason I stopped looking at the strategy. 

However, this did not change the perception that the initial premises on which the strategy was based would remain valid going forward. The strategy acted as if within a rigid black box with a definite start and end date. Going back to 2004, for instance, would not improve performance, just as going forward (pass 2021). The “coincidental” correlation during the simulation's trading period had no reason to prevail outside of it. A simple walk-forward simulation dramatically confirmed that.

Another point of interest (at least for me) was that a strategy should scale up, either by leveraging or increasing its initial stake. At least, that is what the provided equation implied.

Also, there were trading decisions made that would not have been considered in 2008 or 2010-02-09 (TQQQ) for that matter, and since they served as premises to the strategy, it made their use part of some forward biases too. Meaning using knowledge that was not available at the time, or would not have been considered, simply made the strategy a “what if” scenario that might be hard to duplicate going forward. The presented walk forward show that point too.

For me, it was not that the trading signals worked since without them, you did generate more profits. It was simply that in a general uptrend (that lasted from 2009 to 2021) about any trading theory or strategy could have made money without really trying by simply going long most of the time.

As was also said in that post: “A simulation is just to show how your strategy would have behaved, showing you what might have happened if such and such. The important part is what happens after.”

Thanks for following up, Guy Fleury. OK, I see. Interesting. To make it even more interesting, we could add the signals more fully, i.e. going beyond the UUP signal. Moreover, we should consider that the signals only partially determine the algo’s outcome. This is important. In this regard, I recommended that we distinguish between the algo’s three components, outlined above. The dramatic downturn in your previous post’s image is mainly caused by a component that the signals have nothing to do with (out holdings selection). It would be interesting to see whether the algo’s underperformance that you have identified in the years before 2008 is because of:
(a) an incomplete set of signals that is being used, 
(b) an underperforming out holdings component, or 
(c) an incorrect market timer, i.e. underperforming signals.

Good stuff, anyway. Thanks for getting the discussion going.

Hi Peter Guenther ,

I have tried to visualize the signals on a chart in relation to the market. In essence, the signals are all about measuring the risk appetite and "money flow" of the market participants.  I have coded a a simple indicator at TradingView which may capture the essence of what the signals are trying to measure:

Risk Appetite Index (RAI) = (XLK + XLY) / (XLU + XLP)

Here's the chart of the RAI versus SPY:

When the RAI is sloping up, it would mean market participants are bullish and money is flowing into the market.  As can be seen in the chart, the RAI is generally sloping up until the break of long term trendline in Jan 2022.  This may explain why the algo has been performing well until Jan 2022.

I am monitoring the RAI for it start sloping up again, as this could be the signal to put the algo to work again.

Here's the TradingView code for the RAI:

//@version=4

study("Risk Appetite Index", shorttitle="Risk Appetite Index", format=format.inherit, precision=4, overlay=false)

// Add the script's inputs

symbol1 = input(title="Aggresive1", type=input.symbol, defval="BATS:XLK")

symbol2 = input(title="Aggressive2", type=input.symbol, defval="BATS:XLY")

symbol3 = input(title="Defensive1", type=input.symbol, defval="BATS:XLU")

symbol4 = input(title="Defensive2", type=input.symbol, defval="BATS:XLP")

ratio = ((security(symbol1, timeframe.period, close) + security(symbol2, timeframe.period, close))) / ((security(symbol3, timeframe.period, close) + security(symbol4, timeframe.period, close)))

// Plot a line

plot(series=ratio, color=color.red, linewidth=1)

Hello,

Don't know if anyone is still working on this algo, but if you are, you may have noticed that it absolutely craters in 2022. I think it is likely because of inflation, and the unprecedented (until recently) situation in which bonds and equities are selling off in tandem. So, to avoid this, there should be a screener to choose an out-of-market name (such as TBT) the performs during inflation and equity/bond confluence. 

I've just started looking at this, so have no solution to propose yet. Any ideas would be really welcome.

Replying to my own comment here… I apparently overlooked the dialogue of the past month or so. Good ideas expressed, and certainly the walk-forward discussion by Guy Fleury is great insight. One difficult aspect of this kind of stock market strategy modeling is counter-intuitive to some, and that is that longer periods of historical performance don't necessarily lead to better modeling decisions: the walk-forward example show this vividly.

As has been noted, 2008 - 2021 was generally up, up, up, so strategies that used leverage and generally bullish mechanisms are almost guaranteed to perform well in back tests. One reason this happens is that modelers typically define an algo's quality by looking at backtest absolute return as a measure of performance. This isn't necessarily bad, but neither is it necessarily robust to out-of-sample performance..

What is needed is a good analysis (ideally during development) of “What's gonna kill this thing?” One thing that stands out to me is the period's low interest rates, general absence of inflation, and generally negatively correlated Eqiuty/Bond returns during that period. So, a question such as “what if equity and bond returns become positively correlated?” would be a good place to start poking holes in the strategy.

The aspect of the algorithm that seems to me to have opportunity for improvement is the out-of-market selection. The built-in tickers (e.g., TLT, TMF) are long-bond ETFs, which have gotten hammered due to 1) generally, inflation driving up treasury yields; and 2) treasury and equity returns exhibiting positive correlation during 2022. Finding metrics that can select for changes from the in-sample time window seem like a place to go looking. Also, aside from this inflation/correlation issue, what other issues might we be able to identify that could be expected to kill the algo?

Hello Mark, I just joined quantconnect a couple of weeks ago and found this thread and the in/out thread.  I am a beginner at Python and this platform but am very interested in this algorithm.  I have put together a few of the code snippets from various back tests in the two threads to get more of a selection universe for both the in and out holdings with TBT included.  The attached back test uses TQQQ which I know is leveraged and taking advantage of the recent huge gains, but it is fun to see what could have been!  I would like to see this thread keep going and improvements made as well. 

OK completely tangent observation. I noticed that almost 25% of the trades happen in the past year (all loosing trades) for this backtest of about 14 years. 

Any thoughts?

@Glenn, like most, I noticed your strategy version has great numbers. However, what caught my eye was: “...SetLeverage(10)”. I think it is the reason you had a 62.2% CAGR over the 14-year period.

If you reduce leverage to “...SetLeverage(1)”, the CAGR then falls to 2.3%. That says a lot. It is still positive, but then it is not that high.

I found some of your code interesting, especially the impact over the last year. I will look at it more closely. Nonetheless, it seems that the 10x leverage had a major impact there too.

I have never seriously considered putting leverage higher than 1.3x-1.7x on strategies I have worked on, even when using 3x ETFs. However, from now on, I will be looking at 10x leverage as a new upper limit to see if a strategy would blow up if pushed that far. We should know if a strategy could handle from 1x to 10x leverage, even if we do not intend to use it at those higher levels.

If you want to see your strategy fly, do the following: increase weights by 1.3 in “...SetHoldings”, set the initial stake at 1 million, and leave “...SetLeverage(10)” as is.

Anyway, thanks for the heads-up. 

@Peter, what you suggest is like saying: since “now” we know the outcome (the signals did not do so well last year) let's redesign the thing, and “make adjustments” to handle those “anomalies”. That would be part of the process that makes it curve-fitting, obfuscating the fact that during the last year, those who followed the strategy's trading signals, as is, had portfolio meltdowns of -85%. They now need a 667% rise just to break even after such a debacle (saying it will take years to recuperate).

For any trading strategy, what really matters is its live performance and not so much its simulated past. Adjustments made based on after-the-fact data might not measure up so well on future performance either. The market, in general, has been going up for the first 13 years. It even became hard not to make a profit. During those 13 years the strategy might not have needed downside protection, but then, all of a sudden, it badly needed some. And it was not there.

We have to plan for what is coming our way before it happens, not after, since then, it is usually too late, the damage is done. Therefore.

We can all agree a strategy needs to have protective measures. At times, some are built-in, while at other times not. We can modify the strategy and plan for what kind of protection might be needed going forward as in a just-in-case scenario. I usually add them after all the debugging and if the strategy has some merit. I want to see what would have happened pushing the strategy's limits before adding protection. When adding the protection, if the strategy's CAGR breaks down, then the strategy is dropped. But at least, I would then know where the strengths and weaknesses are.

As long as protection is not needed, it will make no difference in the final outcome. However, if something bad happens, like in this case, it would tend to protect a portfolio from major downfalls. This does not mean this strategy is necessarily bad, I only say it needed and still needs protection.

This raises a more fundamental question: were these trading signals any good in the first place? I had the signals at about 80% wrong, and over the last year, they proved to be wrong most of the time. This is like declaring a signal as good if it ends up generating money and bad otherwise. Regardless, another real question is: will those same signals hold going forward? They showed they did not over the past year, will they be better next year and the years after that?
 

no idea why this performs so differently compared to the ETF version of In Out. I thought this was just supposed to have a universe selection for stocks? the other in out algo has been mostly sitting in cash majority of the time, minus a trade here or there. 

It should be having basically the same trade signals - different selection

Also probably helps to have stop losses. 

Hmm ... not really what I am saying, Guy Fleury 😊 
What I am saying is that the unaltered signals (v8; see October last year) provided a relatively accurate indication in terms of when we should have gotten out of the equity market during this recent downturn. See the post below for live trading results that are based on the signals. 
Your point about downward protection is fine.


Interesting approach, Winston Chan, thanks for sharing! I get the basic idea, although let me muse about that a bit more. One quick observation: You seem to directly sum up ETF prices. Using indexed prices could be a cleaner approach. For example, you could divide all prices by their first observation in your time series (or alternatively, the beginning-of-year price for each year). This would make the different ETFs' data more comparable. Might be worth a try to see what happens.


https://www.quantconnect.com/forum/discussion/9632/amazing-returns-superior-stock-selection-strategy-superior-in-amp-out-strategy/p3/comment-41559 

@Peter, here is what I am saying.

You have a strategy that lost -85% of its value from its peak last year after a successful 13-year simulation. No matter how we would want to spin it, the loss is still -85% when it mattered the most (that would have been last year when someone could have traded it live).

Here is the chart for someone that would have started trading that strategy on November 1, 2021. It is self-explanatory.

Is the strategy all bad? I do not think so. Was it bad for live trading in the state it was last November? I would have to say regrettably: YES. Once you lose -80% of your portfolio in a single year, I would not care if at times some of those signals were “coincidentally” right for a while. The final result would still matter and be the most critical of all.

BTW, here are the metrics for the above chart:

There are no numbers in that chart that we could classify as good or enviable. It had a loss rate of 86% and a Sharpe ratio of -1.27. No one, having played this strategy live from November 2021 onward, would say that this is a real moneymaker or that the trading signals were great or mostly accurate.

Thanks Glenn Vinson for sharing your code. I haven't done much research yet, but on first look, I like how you've set up the Growth and Safety selections. I look forward to digging in.