Amazing returns = superior stock selection strategy + superior in & out strategy

Radu Spineanu: Welcome back to the discussion! The data-consuming line is self.SetWarmUp(timedelta(350)) in the algo's initialization section. If you reduce this, e.g. to self.SetWarmUp(timedelta(100)), I reckon this should get you past the runtime error.

Thanks Peter Guenther ! Great to be back. Still trying to wrap my head around QuantConnect.

Hi Peter,

To assist with development of this algorithm, I've added a manual warm up and consolidator for the lookback window. This improves the execution efficiency as we avoid daily calls to the History method.

I've also replaced the `hist_shift` calculation to utilize the built-in `rolling` Pandas method.

Lastly, a new condition has been added to the `trade` method to ensure the algorithm has current market data before placing a trade. This removes an error message that's present in the previous implementation.

See the attached backtest for reference.

Best,
Derek Melchin

Derek Melchin: This is pure magic! Thanks so much for fundamentally overhauling the algo's data engine :) The efficiency gain, I am sure, will be appreciated by everybody using this algo in some capacity (optimization, trading, testing etc.). Very nice work!

So the strategies last trade was on Oct 6th into bonds (I entered mid cycle on Nov 9th).  There has not been a trade since.  I would assume there must be a signal that is keeping the IN from firing.  I have watched the price go up to nearly 15% gain only to come all the way back down.  Perhaps there should at least be a very loose trail stop of some sort.  It really sucks to see all gains evaporate like that.  I was tempted to exit manually as this is real money, but want to stay in sync with the algo.  

Anyway, I know this is a long term type strategy, but I'm tempted to try messing around with a 35 period Linear Regression type trailing stop or 20EMA on the hourly to check closes against in case of trending price.  I'll have to test this in NinjaTrader as I just don't know the tools/indicators on this platform as well. 

I've tried in/out with quality momentum stocks and I'm seeing a similar thing to Nathan, last entered in Oct. This is the code.

Derek Melchin do you spot anything obviously wrong?

Radu Spineanu,

Did you have a chance to complete the full backstest (self.SetStartDate (2008, 1, 1))
of this strategy in the last 4 days?

@Derek, sorry to raise the following but I have some issues.

ARKK starts trading in 2014. Therefore prior to that date, trades are only on the TLT and IEF bond ETFs (first 6 years of the strategy). It kinds of defeats the notion of switching to bonds prior to 2014 since the strategy could only be in bonds in periods identified as “out”.

From 2014 to 2017, ARKK traded on average less than 10k shares per day. Some days with 100 shares! Therefore, that also kind of defeats the switching to stock gambit. It is only this year that the stock's price and volume really surged. Meaning that prior to 2020, the strategy might have days where its market orders would have exceeded the daily traded volume or took a large chunk of it.

My point is no one in 2008 would have chosen ARKK as the sole long candidate for their long-term portfolio since it was not even available. And most probably, only a few individuals would have done so after its 2014 debut, except maybe in 2020 and there only based on hindsight, not before but after seeing its price rise. Again, making it not that good a candidate for this strategy over its past. But that could change going forward...

Other issues, not related to your version, is the playing of the 3x leveraged ETFs. It should be noted that the starting dates for TMF, TYD was 4/15/2009 while TQQQ's starting date was 2/10/2010. There is nothing wrong with using 3x leveraged ETFs as long as we accept that those were the starting dates and that no transactions would have occurred prior to those dates. Using 3x long leveraged ETFs will tend to increase performance without having to pay leveraging fees. However, they are subject to return degradation if held for extended periods. The point would be to use TQQQ once you declare the trend as up (after 2/10/2010 evidently) and exit when the program made those switching to bond decisions.

If, and the if counts, if one wants to use those 3x leveraged ETFs, the starting date cannot be sooner than 2/10/2010 since prior dates will only put you in bond ETFs.

My take:

I raised the stakes to $1M. Used the 3x leveraged ETFs with no added leveraging. Shortened the wait days. Rescheduled the EveryDay() to 5 minutes after market open, and the WeekEnd() to 30 minutes before close. By rebalancing 120 minutes after the market opens on the WeekEnd(), you are missing almost one trade day per week. Nonetheless, TYD was not that good a choice over the period since at times its daily trading volume was quite low. The easy solution would be to simply remove it and only play TMF.

The added performance is mainly due to changing a few numbers in the program and going for the 3x leveraged ETFs. I consider the changes as something trying to maximize the time in the market, first by reducing the wait time on switching and then going for the early entry and the late rebalance on WeekEnd(). It did increase volatility, it did increase max drawdown, but also raised the bar in the profit department. Playing the $100k initial capital scenario should be rewarding as well for those interested. Just with these changes, the strategy had a remarkable 61.68% CAGR! Nonetheless, one can do even better as previously shown...

For those wishing to push on the machine a bit, you might want to use some shorts when switching to bonds. The reasoning behind this is simple. You have something that declares the market as up or down. It has no middle ground. When up, you are in TQQQ, a 3x leveraged ETF. It should coincide with what, technically, you used to declare an upmarket. By using TQQQ, you increase the volatility and beta relative to a market average proxy (QQQ). It tends to help your returns if your upmarket declaration does coincide with an upswing in the overall market. A triple ETF can also produce more downside when your up declaration is wrong. So there is caution to be had. The importance of having a good 'bond' switcher.

The following chart shows when part of the bonds are replaced by a short. It has a 90.137% CAGR over the trading interval. Effectively showing that pushing on the 'out' side of the equation can also help to be even more productive.

Guy Fleury,

What is maximum and current drowdown of this strategy now in USD?

Hi Radu,

The algorithm is stuck in TLT because everytime the `extreme_b` condition is satisfied, the exit day is moved further into the future. We are still waiting for the condition

to be satisfied to signal the exit. See the plot of the backtest below for reference.

Here are some other issues with the algorithm:
-Since we are using Scheduled Events intraday, we need to subscribe to minute data resolution to avoid stale fills.
-Instead of calling AddEquity in `rebalance_when_out_of_the_market` and `rebalance`, we can move this to the Initialize method
-Replace

with

I've resolved these in the backtest below. However, there is further work to do, including:
-Replace the return calculation in `calc_return` with the MomentumPercent indicator.
-Replace the `History` call in `rebalance` with the SimpleMovingAverage indicator.
-Replace the daily `History` call in `record_vars` with a consolidator.

Hi Guy,

Yes, it is important to consider the start dates and liquidity of the securities we're trading. When implementing the revision I posted previously, the traded securities remained the same from a previous version of the algorithm. The algorithm was only changed to resolve the three issues mentioned in my previous post.

Best,
Derek Melchin

Hi Derek!

May I ask you a question about the algo above? Why the stock symbols are not being added automatically by the universe selection and you need to use the line below:

self.SetHoldings(Symbol.Create(symbol, SecurityType.Equity, Market.USA), weight) 

Regards,

Leandro

Yes it seems the treasury bond yield curve signal (SHY) is triggering over and over again.  Idk maybe we just remove the SHY signal altogether?  Fed has printed like what $14T in QE and bonds, etc.  I just don't think they can afford to have yield curve go up and afford to pay off those debts or the 20% of zombie companies surviving only because of 0% interest rate debt.

Anyways, here is the log with the signals and dates...

2020-10-06 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False
2020-10-07 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False
2020-10-08 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-09 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-12 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-20 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-21 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-22 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-23 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-10-26 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-03 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-04 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-06 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-10 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-11 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-12 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-11-13 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False
2020-11-19 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False 
2020-12-02 11:30:00 :    symbol XLI False DBB False IGE False SHY True UUP False G_S False U_I False C_A False

Nathan Swenson I don't know if adding a moving average (SMA, DMA, etc) as an "and" condition of sort would fix this problem. I have tried SMA type crosses and they are just too simplistic for market chop that routinely occurs where you wouldn't want to exit or enter.

I think we have to decide to fix the signal of SHY or not. But more often than not, SHY is the main signal that triggers the OUT condition, so if we remove it will the algo stop working? And I did one more basic test to remove it and the answer was 'yes'.

Nathan Swenson after commenting out the bond yield (SHY) signal, it still performs very very well. And as of 4/24/2020 it enters and stays in the market.

Let me know thoughts. I think SHY and TLT will stay under pressure since the last 30 years they just want to destroy bonds and force all money into equities. Unless they will raise rates for inflation or something I think SHY will keep going lower and lower --- and generate that "fake" Fed created signal which doesnt underscore true equities weakness.

But anyways here is my backtest, commented out SHY bond signal. As aforementioned, as of 4/24/2020 it enters and stays in the market. 1844% return since 2008, not too shabby.

Thank you running those tests Goldie!  Those are some decent results for SPY.  The only issues I see are the increased drawdown and lower PSR score.  I would be hesitant to remove the SHY signal.  Yes it does improve the November results, but I do believe the SHY signal is of importance.  Perhaps it's weighting needs adjustment in the Out decision?  I will have to play around with the signals a bit.  I still don't have a full understanding of the algo despite reading the code.   I need to learn a bit more Python and work through some of the tutorial here at QuantConnect so that I can contribute more. 

So bonds SHY prices and TLT are inversely related. Since 2001 and especially since 2008, the treasury has been killing bond yields to pump up the economy right? Even since Reagan era there has been war on bond yields. So they can't keep pumping the economy if they raise bond rates. The US doesn't produce anything right? So how can they increase bond rates (SHY) and keep pumping equities? Speaking about SHY since the 2008 crisis it is not a good signal, because they will keep rates close to zero percent for a very very long time. Add to that they COVID measures to keep rates at zero for the next two years which the FED has very publicly declared. The algo prints out those true false signals based on 1% deviations in a number of signals like SHY or currency pairs, etc. Unless anybody believes bond yields will spike, SHY needs to be taken out of the formula. Just no longer a good indicator for American fiscal policy until we shift back to a more responsible Volcker era economics or at least until the FED normally raises rates again...because right now SHY yield is non existent.  Using it, post Covid the algo may not fire properly for years.

Hi Guy, 

I tried replacing the "out" bonds with leveraged short QQQ like SQQQ, but I am not seeing nearly the same results.

Is there anything else that I should change?

Raymond, if you look at SQQQ, you'll find that the stock's value depreciates over time.

Raymond Simmons: Thanks for joining the discussion and a valid observation! In line with your observation, when you go to the In & Out algo discussion (page 1) and the Intro (Ctrl + F search for "Intro") there, it notes "What we realized in our earlier discussions is that these alternative assets (bonds) critically contribute to the strategy’s overall return (‘riding the jitter wave’). However, we also found that the ‘out’ indicator is not sufficiently precise, i.e. it should not be used directly to go short on the market."
This is pretty much the gist of what you see in your backtest.
Dan Whitnable performed a similar test to yours on the original algo, going short on the SPY, and noted: "Here is the same algo as above but, instead of going 'out' of the market and into bonds, it goes short SPY using SH. Returns are not nearly the same, but moreover, volatility and drawdown are much higher. If the signals were finding when to get 'out' of the market, one would expect better performance? Just musing, but it almost seems the signals are really just finding when to get into bonds?" (see in the Quantopian discussion archive that Kyle K Oates has kindly shared in the In & Out discussion thread, page 1).
I reckon the quest of finding a set-up that creates a more precise 'out' signal that allows going short on the market is still an open endeavor.