There seems to be quite a difference in the usage of indicators in the following two classes:
EMACrossUniverseSelectionAlgorithm and
My concern with the former is what happens when stocks are added and removed? There does not seem to be any particular provision for warm ups or deletion - this could lead to poor trading and over usage of resources.
The second model uses a method called reset(). The first model does not. Presumably reset() makes up for a gap in a security which keeps coming in and out of play? Do I need to use it? Is this now best practice?
The MACD AlphaModel introduces a new concept:
The consolidator:
This initializes the MACD for each added security and cleans up the indicator for each removed security.
algorithm.SubscriptionManager.RemoveConsolidator(removed.Symbol, data.Consolidator)
My problem is this: it is very difficult to see the full picture using the Online IDE. And very time consuming, requiring a back test to be re-run many times.
Were I drafting indicators on my own machine I would tend to send them to DataFrames as they came and went so that I could see exactly what was going on at different points of time. Or some such routine. I have been rigging up temporary dataframes in the IDE but it is far from ideal and in any event I have not managed to pull the whole indicator history down to inspect it.
It would help therefor if someone could give me a view on the whole matter. Perhaps it is explained in the documentation somewhere but I don't think so.
So here is my question:
Assume a weekly signal from the Universe Selection Model which needs to be processed using indicators in the Alpha Model. What is the best of your demo models to show how indicators should be initialised, mainted/updated and then removed if a stock is dropped in a particular week?
Sorry for the long post. Any advice would be welcomed.
Adam W
Take a look at the examples here and here, with add some code for warm up with a history function (with a property that flags ready in your indicators if it is done warming up) for new securities, and cleaning consolidators for removed securities.
Have you looked into the Research environment? I've been brute force debugging with backtests as well, but in hindsight going through each call line-by-line in a Jupyter notebook is probably more efficient. Community posts (including ones from Jared/QC Staff) and digging through the LEAN source code also tended to be more helpful than the documentation for me.
Adam W
Also the .Reset() method resets indicators to their initial state. Presumably, it is creating a new instance of the object with the parameters defined in def__init__(self).
Anthony FJ Garner
Thanks for those examples - I will go through them.
I don't think one is supposed to use the history function with this kind of frequency - I think you are supposed to use rolling window since it is more efficient. Yes, I have looked at the research environment but you can't call course/fine from it and you can not loop through the data - or at least so I understand. Agreed with posts/source code. I am probably going to have to download Lean to de-bug through the entire stack - the problem is it does not ship with any sample Fundamental data.
The other thing is that the alpha models I quote above do not use either history or the rolling window so I assume they are somehow "warming up" the indicators in another way. I'm not sure how at present, I'll have to go through them with a fine tooth comb. But again you can't use "Update" or "OnSecuritiesChanged" in Research?
As to consolidators, I am only using one resolution. The RSI Alpha Model also uses only one resolution - Daily. So I am not sure what it is consolidating. Perhaps there is some necessity to consolidate the different time periods used in the RSI? I don't know.
All this becomes evident with time. Much time! Too much time! But it is rather like trying to paint a large canvas while looking through a pinhole.
Anthony FJ Garner
None of those Indicator Alpha models I quote above WARM UP indicators.
If you are using the Course Fine Universe with indicators it seems you need to demand History each time a stock comes into play?
What is the best way to warm up indicators on an on-going basis where you are using the Course fine Universe?
Anthony FJ Garner
Indicators
See code provided by above link. Seems we have to use History after all? Very, very odd that those Alpha Models I quote above don't use history. If you have a big universe and long term indicators you are going to run into trouble with those algos. They will simply miss trades (Idicator not ready0 which otherwise could have been captured.
See code provided kindly by Alexandre Catarino:
from Selection.FundamentalUniverseSelectionModel import FundamentalUniverseSelectionModel
#https://www.quantconnect.com/forum/discussion/7132/how-to-access-history/p1
class VentralModulatedThrustAssembly(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2019, 7, 9) # Set Start Date
self.SetCash(100000) # Set Strategy Cash
self.SetUniverseSelection(SelectionModel())
self.SetAlpha(ConstantAlphaModel(InsightType.Price, InsightDirection.Up, timedelta(1), 0.025, None))
self.SetPortfolioConstruction(EqualWeightingPortfolioConstructionModel())
class SelectionModel(FundamentalUniverseSelectionModel):
def __init__(self,
filterFineData = True,
universeSettings = None,
securityInitializer = None):
super().__init__(filterFineData, universeSettings, securityInitializer)
self.periodCheck = -1
self.coarse = dict()
def SelectCoarse(self, algorithm, coarse):
if algorithm.Time.year == self.periodCheck:
return Universe.Unchanged
self.coarse = {x.Symbol: x
for x in coarse if x.HasFundamentalData and x.Volume > 0 and x.Price > 0}
return [symbol for symbol,_ in self.coarse.items()]
def SelectFine(self, algorithm, fine):
DERatio = lambda x: x.OperationRatios.LongTermDebtEquityRatio.ThreeMonths
fine = [x for x in fine if DERatio(x)
and x.CompanyReference.CountryId == "USA"
and (x.CompanyReference.PrimaryExchangeID in ["NYS", "NAS"])
and (algorithm.Time - x.SecurityReference.IPODate).days > 180
and x.EarningReports.BasicAverageShares.ThreeMonths *
x.EarningReports.BasicEPS.TwelveMonths *
x.ValuationRatios.PERatio > 1.52e8]
if len(fine) == 0:
return Universe.Unchanged
self.periodCheck = algorithm.Time.year
sortedByDEratio = sorted(fine, key=DERatio, reverse=False)[:50]
symbols = [x.Symbol for x in sortedByDEratio]
averages = dict()
history = algorithm.History(symbols, 200, Resolution.Daily).close.unstack(0)
for symbol in symbols:
# Remove NaN: symbol does not have 200 daily data points
df = history[symbol].dropna()
if df.empty:
continue
mom = Momentum(126)
for time, close in df.iteritems():
mom.Update(time, close)
# Adds Momentum to dict only if it is ready
if mom.IsReady:
averages[symbol] = mom
# Update with current data
for symbol, mom in averages.items():
c = self.coarse.pop(symbol, None)
mom.Update(c.EndTime, c.AdjustedPrice)
sortedbyMomentum = sorted(averages.items(), key=lambda x: x[1], reverse=True)
return [x[0] for x in sortedbyMomentum[:5]]
Adam W
As to consolidators, I am only using one resolution. The RSI Alpha Model also uses only one resolution - Daily. So I am not sure what it is consolidating.
You should use Minute data and then consolidate them into Daily bars.
The Algorithm Framework is a helpful starting point in cleanly separating modules, but I've found that working with a more classical approach gives me more flexibility and control over what's going on without having to spend an hour digging through LEAN source code. I bypass the whole AlphaModel construction, and instead my Alpha model is defined in a function which is called in OnData() when certain criteria match (i.e. a flag for model is ready, time rules, entry/exit, etc), and any heavy computations are done on a scheduling function before markets open. This is more similar to the architecture of Quantopian, maybe give it a try.
Anthony FJ Garner
Adam W, I wholly agree with your point of view. The classical model is along the lines I have always favoured in my own approach - a direct and simple route to what it is you want to achieve. At present however I am concentrating on the algorithm framework for a number of reasons.
Not least of which is that that is the direction Lean providers seem to want to go. In which case that is where their future development efforts will presumably be directed.
Anthony FJ Garner
But the big question here is WARMING up indicators for the course/fine universe And this has been answered. You either use code similar to the above provided by Alexandre or you wait until the new warm up code quoted here by Jared is extended to all indicators and thoroughly tested:
self.EnableAutomaticIndicatorWarmUp = True
Not implemented for all indicators yet see Github.
Jared Broad
We will always keep support for classic style algorithms as they are more suited for individual accounts. The framework is the architecture for a full firm in a single algo. I hope as the framework matures it's benefits will become 10x the classic style even for single accoounts
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Adam W
Absolutely, the benefits of the framework speak for themselves. I implement features of the framework architecture whenever possible, but for any persistent errors/bugs I simply work around it manually rather than spend hours trying to debug it with my limited C# knowledge. When the framework matures more and when one gets more accustomed to the API, I'd imagine it'll make the whole process significantly smoother for everyone.
Mark Reeve
Great thread. I second the concerns about lack of WarmUp within the docs for AlgoFramework. Even in the Bootcamp tutorials they failed to warmup the MOM indicator? And so, I assumed when using Indicators in AlphaModels there was no need to WarmUp. I have since learned the hard way, that is not the case!
Rahul Chowdhury
Hey Mark,
Thank you for the feedback on the documentation! The 200-50 EMA Boot Camp describes how to warm up indicators with historical data using the framework.
Best
Rahul
Aalap Sharma
How would this example work for the Bolliger bands and MACD ? any examples somewhere
Would be something like
def WarmUpIndicators(self, history): for index, row in history.loc[str(self.symbol)].iterrows(): self._macd.Update(index, row["close"]) self._bb.Update(index, row["close"]) if self._macd.IsReady and self._bb.IsReady: self._macdHistogram.Add(self._macd.Histogram.Current.Value) self._bbupwindow.Add(self._bb.UpperBand.Current.Value) self._bbmidwindow.Add(self._bb.MiddleBand.Current.Value) self._bblowindow.Add(self._bb.LowerBand.Current.Value)
Shile Wen
Hi Aalap,
I am unsure what your question is. The 200-50 EMA BootCamp mentioned by Rahul should be sufficient, except only the indicators are changed.
Best,
Shile Wen
Aalap Sharma
Thanks I think I figured it out. I wanted to warmup the LowerBand and UpperBand values as well for BB indicator
Anthony FJ Garner
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!