Share New Research 
Hi there,
I'm getting "Runtime Error: Algorithm took longer than 10 minutes on a single time loop" in the algorithm that does nothing but universe selection and runs for just one day. I'm attaching the source code.
The algorithm needs to warm up some indicators, so it performs a History call inside the coarse universe selection function. This History call is rather sizable: over 7K symbols, 30 daily bars each. It runs for around 6 minutes and returns successfully.
The fine universe selection function returns just a 100 symbols. It runs fine.
There is a function scheduled to run 10 minutes prior to market open. It runs okay.
Then the algorithm just hangs. It sits there for 10 minutes and exits with the Runtime Error. If I register another scheduled function to be called, say, 1 hour after the market open, it is never called.
Here is the log output:
05:53:58: Launching analysis for ...
06:00:16: Warmed up 7329 stocks
06:00:16: Fine collection is 5287 long
06:00:16: Today's universe: OPHT, PRTO, FFHL, CERC, CVM, ANY, RXII, ARDM, CLSN, CETC, RGSE, HTBX, NVLS, ARWR, SHIP, RVP, PACB, CLSD, DCTH, SCON, GEVO, XGTI, PFMT, OVAS, INOV, OHRP, OSG, FOLD, NURO, ATOS, ROSG, XBIT, FNCX, GALE, LIFE, CYTR, ONS, OHAI, YECO, MBOT, AIRG, ESES, BAS, ORPN, ATEC, MBRX, MGNX, MACK, CLRB, CTRV, VHC, ABEO, JAKK, HOTR, CAPN, SHLD, DTEA, ALBO, IFON, SIF, DXTR, BLCM, ABIL, IBIO, PLX, FCSC, RPRX, TEAR, OBLN, IMUC, QTM, PSIX, STAF, AMRS, NVIV, APHB, AGIO, WVE, XLRN, DLTH, PTX, SYMX, TVIA, PRKR, SBBP, MYOK, SELB, FTK, ETRM, RDUS, INSY, MXPT, STRM, ALN, FOSL, RKDA, LONE, PN, CLBS, TROV
06:01:40: before_trading_start: 51
06:01:40: Data for symbol DCIX has been limited due to numerical precision issues in the factor file. The starting date has been set to 7/4/2017.
06:02:03: Data for symbol DRYS has been limited due to numerical precision issues in the factor file. The starting date has been set to 1/21/2017.
06:10:20: Runtime Error: Algorithm took longer than 10 minutes on a single time loop.
Weird things:
- before_trading_start function logs the size of Securities collection. For some strange reason, it is 51. This number seems to have nothing to do with the size of the universe. In the attached code it is 100, but if I change it to 10, the size of Securities collection is still 51 at this point.
- Those "Data for symbol ..." warnings complain about the symbols that are not in the universe. What's going on?
- In order to perform that History call in the coarse universe selection function, I have to call AddEquity() for each of the symbols. Initially, I was just calling RemoveSecurity() on them after I was done, but I noticed that "before_trading_start" was reporting that the Securities collection length was 7K+. So I added an additional Securities.Remove() call which brought the size of the Securities collection to this weird 51.
- If I pre-filter the coarse collection prior to the warm up step, it helps. This is not an option for me however since I must have the indicators calculated for the broadest collection of symbols.
- If I comment out the lines 42 and 43 ("self.AddEquity" and "self.Schedule.On") it also helps. If I leave the line 42 intact and comment out only the "Schedule" line, it would hang still. Pretty weird.
Jing Wu
self.AddEquity(symbol, resolution) request the equity data for the specified symbol. In universe selection algorithm, you don't need self.AddEquity().
The self.CoarseSelectionFunction and self.FineSelectionFunction are set to triggered at 00:00 every day by default. In the current time loop, the data of symbols returned in FineSelectionFunction(self, fine) are subscribed in OnData(self, data) automatically, symbols not returned in this time loop by FineSelectionFunction(self, fine) will be removed automatically.
You didn't specified the resolution in AddEquity(), the default resolution is minute. Therefore, you're actually subscribe all data for over 7k symbols with minute resolution. It will slow down the algorithm significantly. You should remove AddEquity() in the algorithm except the SPY one.
For large history requests, once you get over about 120K data points you'll run into timeout issues. For reference, coarse data set is ~7K symbols, so even 30 days at daily resolution on all symbols is 210K data points.
More reasonable would be trimming the result set down using coarse selection and then issuing a single history requests for the results of coarse to initialize the indicator.
Anton Voskresenskiy
Thanks, Jing,
I was under the impression that I needed to call self.AddEquity() prior to calling self.History() at all times. It appears I was wrong and the self.History() call works fine without it.
One observation, however: previously, when I was calling self.AddEquity(), the self.History() call was returning a DataFrame with Symbol.Value ("A" for instance) as its first level index. Now that I'm calling self.History() without a prior self.AddEquity(), I'm getting Symbol.ID as the first level index ("A RPTMYV3VC57P" for instance). I wonder what causes this discrepancy and what are the rules of thumb here?
And for the future reference: my original code suggests that a sequence of self.AddEquity() + self.RemoveSecurity() + self.Securities.Remove() doesn't seem to fully unregister a stock. I was still having at least 51 of them left registered somewhere and they were slowing the algorithm down. Is there a guaranteed way to unregister a stock once and for all after a self.AddEquity()?
Jing Wu
For history request, you can either explicitly use a symbol object to make the history request or ensure the symbol has been added using the AddEquity() method before making the history request. Therefore, if you request the history without AddEquity(), the symbol is a symbol object. For pandas dataframe in Python, implicit conversion from symbol object to a string ticker is not support, the symbol here like "A RPTMYV3VC57P" is actually a string format of symbol object. When you extract the values for a specified symbol, you should use str() to convert the symbol object to the string. For example
symbol_objects = [stock.Symbol for stock in coarse]history = self.History(symbol_objects, 10, Resolution.Daily)
history.loc[str(symbol_objects[0])]
Please see Alex's explanation in this post.
"self.RemoveSecurity(Symbol)" will remove the data subscription for that symbol. However, The self.CoarseSelectionFunction and self.FineSelectionFunction are set to trigger at 00:00 every day by default. Once there are symbols returned in self.FineSelectionFunction, the data will be subscribed automatically in OnData() for those symbols.
If you want to remove the data subscription for some symbols, you should also control the returned symbols in self.FineSelectionFunction.
Anton Voskresenskiy
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!