When should I launch my live-trading algorithm?

Hi Li,

Me again.

Short answer: I don't know. Maybe others will reply directly to your question. But what you are saying re historical data update frequency is a bit concerning…

Long answer: You can use historical data to warm up your indicators but this is not my preferred way (and the recommended one for, say, advanced LEAN usage). The best way IMHO is to register your indicators for updates, with RegisterIndicator(). Then, if you set SetWarmUp(), your indicators will be warmed up according to their warm up period. If you have custom indicators (ones you programmed yourself), they will need to return an appropriate warm up period int. See 

And you can also check this out: 

I'm only guessing that with registration and warm up, indicators get updated with 1) enough data according to their warm up period and 2) the most recent data - and not data that ends “2 days ago”. But I have never thought about explicitly checking that.

Fred

Hi Fred, I appreciate your response, it's awesome that you're helping people out! 

I guess the goal of this thread is to answer this main question: When does QuantConnect update its minute resolution historical data after Friday's trading day? Is it on Saturday or on Monday when the market opens? 

Hi Li,

https://www.quantconnect.com/docs/live-trading/universe-selection

According to my understanding of this documentation the historical data is updated once the market closes completely at 21:00 EST.

I looked through a lot of posts and documentation for this and this is the only answer I could find…

Hope it helped!

Hi @Li Cheng, for backtesting we update the data ASAP, generally within 12 hours but some datasets are published 48 hours later. Occasionally vendors publish data a few days late, or there is an issue in the backtest data distribution stack, and there isn't much we can do about it. This only impacts backtesting on “yesterday” data.

Our live environment uses a “stream caching system” for recent dates which gives us about a week buffer – i.e. if there is more than a week delay in data processing we could have gaps in live trading, but that hasn't ever happened to date. We also have redundant sources for universe selection so if one source fails the second source covers it. 

TLDR on your original question:

Universe selection is done on deployment, or if Friday then it's delayed until later in the day Sunday so we have plenty of time in case there are any weekend issues. 

@Mak K, thanks for answering my question, you're doing great work for the community!

@Jared Broad, really appreciate the thorough response! My algorithm depends heavily on “yesterday" data because my indicators use minute resolution data to warm up (I'm working with equities by the way), since you mentioned that history data might occasionally be published 48 hours later, would you suggest that I warmup my indicators with live data instead of making a history call during the weekend for Friday's data to warm them up?

Hi Li,

If I may, the 48 hour potential delay is only for backtesting, as I understood from Jared's post. Once you go live, the chances of any delays are very low so your indies will be properly warmed up.

If it is critical, you may add code in your algo to verify that your indies are warmed up using “yesterday" data and log if not (and/or maybe even stop if it is that critical). Live, it should not trigger. In backtesting, you could then maybe backtest with other equities in the meantime (while waiting for the recent data to be added to the backtest dataset).

Fred

Hi Fred, that's great advice! I haven't thought about checking whether it's “yesterday” data before using them to warmup my indicators!

I think I didn't fully understand what Jared meant by a “stream caching system”, does it mean that QuantConnect has a secondary storage for recent historical data that can only be used for live-trading and not backtesting?

In short, using an example to summarize what I fully understood from this thread: 

• If I'm using a 15-minute MACD indicator, I should just deploy my live-trading algorithm on Sunday evening EST, since all of the historical data should be ready at that point.
• If I have universe selection, then universe selection will run immediately after deployment
• So basically, my algorithm will be fully initialized and ready to go before the market opens on Monday