QUANTCONNECT COMMUNITY
This thread is meant to continue the development of the In & Out strategy started on Quantopian. The first challenge for us will probalbly be to translate our ideas to QC code.
I'll start by attaching the version Bob Bob kindly translated on Vladimir's request.
Vladimir:
About your key error, did you also initialize UUP like this?
Share New Research 
self.UUP = self.AddEquity('UUP', res).Symbol
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
QuantConnect™ 2024. All Rights Reserved
|
|
Peter Guenther
T Smith: Echoing Vladimir's post, very well done and thanks for sharing this Momentum/In & Out strategy! This is interesting insights for all, particularly I think Joseph Kravets was musing about a rotation strategy between multiple ETFs. Great stuff!
Joseph Kravets
Thanks Peter! I am starting to think that this could be overfitted but it seems effective.
Chak
To throw a wrench into the conversation, you'll notice that in 4/2013, 7/2020, and a few dates in 2015/2016 (I forget which), you'll find that the algorithm switches out of in_market stocks to out_market stocks. My perspective might be a little contraversial, but I want to argue that the reason why SPY continued going up during these times despites multiple signals firing to exit the market is that the government and/or financial institutions were pumping the market - in 4/2013, it's the start of the economic recovery from the housing bubble, and the federal government allowed/incentivized banks to sell foreclosed houses for well below market, and in 8/2020 it was announced that 3 financial institutions were intentially pumping up the market (Nasdaq, specifically) to continue to go up.
It's just a thought and a different perspective to contribute to the group.
Vladimir
Here is my DUAL MOMENTUM-IN OUT v2
Peter Guenther
Chak: Absolutely. I am wondering whether there might be an additional (hidden) indicator here that could be useful to overrule the algo's 'out' indication in certain situations. Eg similar to the notion of "don't fight the Fed", for which an indicator might be the size of the Fed's balance sheet or, ideally, the announcement of a planned expansion of the Fed's balance sheet (= pumping money into the market). Maybe such data could be matched in via a custom factor/uploaded data or via news scraping.
Goldie Yalamanchi
Peter Guenther yes that must be figured out. All the variations of the algo are great, but those that include SHY as a signal have effectively stopped re-entering the market remaining in `OUT` state since 10/6/2020 and the SP500 has made a 9% gain since then, while the algo has sat out for almost 70 days. Granted there was a lot of chop, but I wonder if the algo will even trigger to back 'IN' state this year or for months into next year. There is another FOMC meeting this week, generally they will say.... we will keep rates low and SHY will keep the algo OUT (my prediction of course).
Peter Guenther
Goldie Yalamanchi: Yep, the SHY is a tricky one. Technically, it is not the Fed's low/zero interest policy/cuts that is making the SHY drift down (the algo is looking for that downdrift that falls into the bottom 1 percentile over a three month window). When the SPY drifts down it means that the market expects an increasing interest rate (= that the Fed raises the rate). A falling bonds price means a larger interest yield. The algo interprets this as a bad thing = increase in the costs of capital. And usually this would be a bad thing for equities. However, the current root cause behind the increasing interest rate expectations is actually something that is good for equities: the market expects the economy to rebound due to the latest Covid vaccine developments. So, we have two forces at play here, a positive (economic rebound) and a negative (economic rebound causing the Fed to increase rates at some stage increasing the cost of capital). Currently, the positive outweighs the negative. Yet this can flip when the market feels that the rebound (i.e. the positive side) is sufficiently priced in.
I reckon if we could disentangle investors' improving economic outlook, we might have a counter-signal that could be used to mute the SHY signal in certain situations in which it should actually be interpreted as something that is the symptom of a positive development.
Vladimir
Peter Guenther,
The latest IN-OUT strategy has 12 information sources.
The logic of decision-making is
if any one of your 12 top advisors says "out" but 11 others say "in"
the strategy should go "out".
Why you choose that rule?
Why one, not seven what is more logical?
Where in human activity you have seen such decision-making rule?
Think about it instead of looking what is wrong with "SHY".
Technically, there is nothing wrong with "SHY" chart.
The problem is in decision-making rule.
Jack Pizza
Hey Peter Guenther
Tried running this live but it doesn't enter trades, commented out warmup period, set self_bein to 1
""" Based on 'In & Out' strategy by Peter Guenther 4 Oct 2020 expanded/inspired by Tentor Testivis, Dan Whitnable (Quantopian), Vladimir, and Thomas Chang. https://www.quantopian.com/posts/new-strategy-in-and-out https://www.quantconnect.com/forum/discussion/9597/the-in-amp-out-strategy-continued-from-quantopian/p1 """ # Import packages import numpy as np import pandas as pd import scipy as sc class InOut(QCAlgorithm): def Initialize(self): self.SetStartDate(2008, 1, 1) # Set Start Date self.SetCash(100000) # Set Strategy Cash self.UniverseSettings.Resolution = Resolution.Daily res = Resolution.Minute # Feed-in constants self.INI_WAIT_DAYS = 15 # out for 3 trading weeks # Holdings ### 'Out' holdings and weights self.BND1 = self.AddEquity('TLT', res).Symbol #TLT; TMF for 3xlev self.BND2 = self.AddEquity('IEF', res).Symbol #IEF; TYD for 3xlev self.HLD_OUT = {self.BND1: .5, self.BND2: .5} ### 'In' holdings and weights (static stock selection strategy) self.STKS = self.AddEquity('QQQ', res).Symbol #SPY or QQQ; TQQQ for 3xlev self.HLD_IN = {self.STKS: 1} # Market and list of signals based on ETFs self.MRKT = self.AddEquity('SPY', res).Symbol # market self.PRDC = self.AddEquity('XLI', res).Symbol # production (industrials) self.METL = self.AddEquity('DBB', res).Symbol # input prices (metals) self.NRES = self.AddEquity('IGE', res).Symbol # input prices (natural res) self.DEBT = self.AddEquity('SHY', res).Symbol # cost of debt (bond yield) self.USDX = self.AddEquity('UUP', res).Symbol # safe haven (USD) self.GOLD = self.AddEquity('GLD', res).Symbol # gold self.SLVA = self.AddEquity('SLV', res).Symbol # vs silver self.UTIL = self.AddEquity('XLU', res).Symbol # utilities self.INDU = self.PRDC # vs industrials self.SHCU = self.AddEquity('FXF', res).Symbol # safe haven currency (CHF) self.RICU = self.AddEquity('FXA', res).Symbol # vs risk currency (AUD) self.FORPAIRS = [self.GOLD, self.SLVA, self.UTIL, self.SHCU, self.RICU] self.SIGNALS = [self.PRDC, self.METL, self.NRES, self.DEBT, self.USDX] # Initialize variables ## 'In'/'out' indicator self.be_in = 1 #initially, set to an arbitrary value different from 1 (in) and 0 (out) ## Day count variables self.dcount = 0 # count of total days since start self.outday = 0 # dcount when self.be_in=0 ## Flexi wait days self.WDadjvar = self.INI_WAIT_DAYS # set a warm-up period to initialize the indicator #self.SetWarmUp(timedelta(15)) self.Schedule.On( self.DateRules.EveryDay(), self.TimeRules.AfterMarketOpen('QQQ', 65), self.rebalance_when_out_of_the_market ) self.Schedule.On( self.DateRules.WeekEnd(), self.TimeRules.AfterMarketOpen('QQQ', 65), self.rebalance_when_in_the_market ) def rebalance_when_out_of_the_market(self): # Returns sample to detect extreme observations hist = self.History( self.SIGNALS + [self.MRKT] + self.FORPAIRS, 252, Resolution.Daily)['close'].unstack(level=0).dropna() hist_shift = hist.apply(lambda x: (x.shift(65) + x.shift(64) + x.shift(63) + x.shift(62) + x.shift( 61) + x.shift(60) + x.shift(59) + x.shift(58) + x.shift(57) + x.shift(56) + x.shift(55)) / 11) returns_sample = (hist / hist_shift - 1) # Reverse code USDX: sort largest changes to bottom returns_sample[self.USDX] = returns_sample[self.USDX] * (-1) # For pairs, take returns differential, reverse coded returns_sample['G_S'] = -(returns_sample[self.GOLD] - returns_sample[self.SLVA]) returns_sample['U_I'] = -(returns_sample[self.UTIL] - returns_sample[self.INDU]) returns_sample['C_A'] = -(returns_sample[self.SHCU] - returns_sample[self.RICU]) self.pairlist = ['G_S', 'U_I', 'C_A'] # Extreme observations; statist. significance = 1% pctl_b = np.nanpercentile(returns_sample, 1, axis=0) extreme_b = returns_sample.iloc[-1] < pctl_b # Determine waitdays empirically via safe haven excess returns, 50% decay self.WDadjvar = int( max(0.50 * self.WDadjvar, self.INI_WAIT_DAYS * max(1, np.where((returns_sample[self.GOLD].iloc[-1]>0) & (returns_sample[self.SLVA].iloc[-1]<0) & (returns_sample[self.SLVA].iloc[-2]>0), self.INI_WAIT_DAYS, 1), np.where((returns_sample[self.UTIL].iloc[-1]>0) & (returns_sample[self.INDU].iloc[-1]<0) & (returns_sample[self.INDU].iloc[-2]>0), self.INI_WAIT_DAYS, 1), np.where((returns_sample[self.SHCU].iloc[-1]>0) & (returns_sample[self.RICU].iloc[-1]<0) & (returns_sample[self.RICU].iloc[-2]>0), self.INI_WAIT_DAYS, 1) )) ) adjwaitdays = min(60, self.WDadjvar) # self.Debug('{}'.format(self.WDadjvar)) # Determine whether 'in' or 'out' of the market if (extreme_b[self.SIGNALS + self.pairlist]).any(): self.be_in = False self.outday = self.dcount if self.dcount >= self.outday + adjwaitdays: self.be_in = True self.dcount += 1 #self.be_in = True # for testing; sets the algo to being always in # Swap to 'out' assets if applicable if not self.be_in: self.wt = {**dict.fromkeys(self.HLD_IN, 0), **self.HLD_OUT} # Only trade when changing from in to out for sec, weight in self.wt.items(): cond1 = (self.Portfolio[sec].Quantity > 0) and (weight == 0) cond2 = (self.Portfolio[sec].Quantity == 0) and (weight > 0) if cond1 or cond2: self.SetHoldings(sec, weight) self.Plot("In Out", "in_market", int(self.be_in)) self.Plot("In Out", "num_out_signals", extreme_b[self.SIGNALS + self.pairlist].sum()) self.Plot("Wait Days", "waitdays", adjwaitdays) def rebalance_when_in_the_market(self): # Swap to 'in' assets if applicable if self.be_in: self.wt = {**self.HLD_IN, **dict.fromkeys(self.HLD_OUT, 0)} # Only trade when changing from out to in for sec, weight in self.wt.items(): cond1 = (self.Portfolio[sec].Quantity > 0) and (weight == 0) cond2 = (self.Portfolio[sec].Quantity == 0) and (weight > 0) if cond1 or cond2: self.SetHoldings(sec, weight)Peter Guenther
Elsid Aliaj: One possible explanation: when the in_out indicator (self.be_in) is 1 (= says "in"), the algo waits until Friday to actually go in ('lazy trader approach'; I only reshuffle/enter equities Fridays). This happens via the second scheduling function (line 74 where it says "self.DateRules.WeekEnd()"). Another possibility is that the in_out indicator says "out" but we are past the scheduled time for the out reshuffling, which in your code happens every day at 10:35 (9:30 + 65 minutes; lines 66-70). Also in this case the algo would do nothing today and only trade tomorrow. Not sure whether this helps to identify the issue (?).
Chak
I'll hint on several things that can be done. Peter introduced the concept to quantconnect, I personally would like him to submit it as an alpha. Here are a few things I've tried:
1) If you look at the IN-OUT Plot function, it is possible to disentangle each signal's individual contribution to the algorithm's decision to exit the market. I see at least 3 ways each signal can contribute:
#1 in the extreme_b filter function that sets a hard boundary
#2 the individual contribution of each ETF to their signal
#3 in the covariance matrix to assess whether the recent signal falls with the 1%.
If you create a second plot similar to IN-OUT but have each ETF represent a line, you can see these weights.
2) if you were to omit other signals besides SHY and at each level in the decision-making process mentioned in #1, you'll find find something interesting.
3) as everyone have mentioned, yes, the model is overfitted. What can we do to resolve model overfit issues besides simply removing variables based on PSR, Sharpe Ratios, and return %?
If we can solve the below two issues, we'd be a step closer to a better model:
#1 Are there methods to set the weights of each ETFs' impact on both the 'IN' and 'OUT' strategy, while addressing the model's flexibility to adjust to future events (i.e., out of sample data)?
#2 Can we further use these weights to create a more refined strategy to include different IN/OUT Market scenarios? For example, are there instances there it's better to use TQQQ instead of QQQ when going back into the market, or to use GLD/UGL and TLT/UGL when going out of the market or you're already out of the market?
Jack Pizza
Peter Guenther Ok i'll just use the prior version i was using, but this line threw an error last time, not sure if anyone knows what the issue is, specifically close.
self.SIGNALS + [self.MRKT] + self.FORPAIRS, 252, Resolution.Daily)['close'].unstack(level=0).dropna()
Runtime Error: LiveTradingRealTimeHandler.Run(): There was an error in a scheduled event EveryDay: QQQ: 120 min after MarketOpen. The error was KeyError : 'close' Stack Trace: System.Exception: LiveTradingRealTimeHandler.Run(): There was an error in a scheduled event EveryDay: QQQ: 120 min after MarketOpen. The error was KeyError : 'close'Maybe it didn't have the closing data of QQQ that day? But it threw the error during 11:30 when it was calculating whether to stay in or out.
Nathan Swenson
Elsid, I was running live and rather than throwing an error, it simply disabled. When I logged in I simply had nothing running live. I'll have to check if there's a way to check for errors.
Peter Guenther
Elsid Aliaj and Nathan Swenson, thanks for raising the issue! From Elsid's error message it seems that the history call did not result in the usual table with a 'close' column (containing the closing prices). Yet, not entirely sure what happened. Derek Melchin probably would know directly. He has re-written the code using a consolidator logic and this might be a way to avoid the error (?).
Peter Guenther
Not sure whether this can solve the issue but the consolidator version is in the thread Amazing returns = superior stock selection strategy + superior in & out strategy and search for Derek's message "To assist with development of this algorithm, I've added a manual warm up and consolidator for the lookback window. This improves the execution efficiency as we avoid daily calls to the History method." Yet it's probably best to wait for Derek's comment since there may be specifics concerning live trading.
Peter Guenther
Elsid Aliaj: Just attaching here the latest version of the In & Out that I got. Much indebted to Derek and other contributors. It includes the consolidator approach for greater efficiency and code that ensures that sells are submitted/filled before buys. Can't promise that this will address the live trading issue but the features may be useful nonetheless for further testing and live trading.
Jack Pizza
Peter Guenther thanks i'll give it a go let you know if any issues arise
Nathan Swenson
Strategy when back into equities today (12/18). Prior trade was 10/6.
Peter Guenther
First crack on disambiguating the DEBT signal
Following up on DEBT (SHY) possibly being an ambiguous signal; see above the notion of an expected economic rebound causing investors to anticipate the Fed to increase rates at some stage, raising cost of capital expectations. We then said “if we could disentangle investors' improving economic outlook, we might have a counter-signal that could be used to mute the SHY signal in certain situations in which it should actually be interpreted as something that is the symptom of a positive development”.
Attached is an implementation that uses increasing input prices (METL and NRES, above median) to mute the DEBT signal.
Peter Guenther
Chak: Good set of questions and insights there, keep on hacking ;)
Nathan Swenson: Thanks for the live trading update! Fingers crossed that the market does not go down, now that the algo has decided to go back in. Then it would have gotten the rhythm of this cycle wrong.
Tentor Testivis
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!