Share New Research 
Hey there!
Yesterday we launched a cool new feature called Live Reconciliation. Sometimes there can be minor issues in code which cause slightly different behavior between live-trading and backtesting. We built live-reconciliation to help identify those issues.
Each time you go live-trading we will run a backtest with the same code as was deployed live. We gather these backtest equity curves together and have added them as a light-gray line on your equity chart. This represents how your code performed in a backtest over the same period.
We are calculating a score representing the error-rate between these curves which should give you an idea of the error rate. The score is based on the "dynamic time warp" between the curves and will be the average daily percent error. Soon we'll display this score on the charts to help you understand the error rate, and have a notebook publishing this research.
There is one known issue: if you've got existing holdings in your live account, and it's not modeled into your backtests; there is no way for the live-algorithm to reconcile that backtest. It assumes starting from no holdings on each deployment. If it impacts many people we may look into parsing the logs to start reconciliation backtests with the live holdings.
Live reconciliation is one step closer to having our backtests perfectly model live trading. We hope as QuantConnect's platform approaches perfection our backtests will be so accurate you could completely trust backtest results! =)
Best
Team @ QC
Josh M
for some reason my live & listed alpha just has a 'out of sample backtest' of 0 -- not sure if this will fix itself over time or if I would have to stop and start the alpha for it to 'fix' itself.
Interesting concept though.
Jared Broad
Hey Josh - there will for sure be alot of edge cases we're going to need to go through one by one. Please drop a message to support with the link to the alpha -- or if its for a live project -- the live project attached and we can debug. If you'd like it disabled you can click on the chart labels to hide it, the chart labels are in the top-left now.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Hugh Todd
A really nice feature. I stumbled upon it Friday afternoon and wondered how long it had been there.
It helps answer those questions about how well live trading performs relative to backtests.
Hugh Todd
Jared Broad When first released the Live Reconcilliation numbers were very close to my live trading. Since that time there has been a definite divergence. Are there any known issues with this? Thanks,
Rahul Chowdhury
Hey Hugh,
The divergence of out of sample backtesting results from live trading results occur due to differences between backtesting data and live data. Backtesting data is clean and well aggregated data, while live data comes directly from the exchange. Backtesting results also depend on fill models, which may not always directly reflect fills in live trading.
We are alway working towards improving Lean's backtesting to more accurately represent live trading. Level 1 equity quote data, which was recently released, takes us another step in that direction.
You can use live reconciliation as a way to check how trust worthy your backtest results are. If there is a large divergence in live reconciliation, it means that the backtest results for that strategy are unlikely to have happened if it had been live traded over that same time period.
Best
Rahul
Hugh Todd
Rahul Chowdhury - Thanks for the reply and I appreciate the work you all are doing to make QuantConnect even better. To maybe assist that, let me offer some numbers from what I am seeing. I believe Jared introduced this on Feb 19. At that time my live backtest and trade account matched within a few hundred dollars. Through March 11 one of those is up significantly while the other is down over 14%. That is a pretty big divergence over a 3 week time period. If I can offer anything more please let me know.
Hugh
Jared Broad
Hey Hugh! This is exactly what we hoped it would uncover, but at this point, it's up to you to dig in and understand why it doesn't line up. We're writing some documentation now on the common reasons why the reconciliation will be different, we hope this will help.
If you think they should be lining up we can assist with a support ticket submitted via the IDE with the live deployments attached.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Hugh Todd
Jared Broad - Thanks. A real backtest over that period matches my trade account very closely so the error is with the reconciliation line. As long as the backtest and the real trading account match I'm good and can turn off the reconciliation line. i just thought someone might want to know.
Hugh
Hugh Todd
Jared Broad - I just realized the problem. My backtest (and live system) pulls in data from outside QC and that data is likely not available to the reconciliation. My apologies for raising this issue and please feel free to delete the above messages so they won't confuse anyone.
Hugh
Jared Broad
Awesome! Glad you caught it! That will be one of the things we put into the common reasons for differences =)
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Laurent Crouzet
Jared Broad , might I propose to manage differently the dates used for "Live Reconciliation" track record?
Each saturday and each sunday (+ each Bank holiday), there is a dataplot for the "Live Trading Equity" (with the same value as the previous day(s)), while the Out Of Sample (OOS) values do not take the week-end data into account, nor the bank holidays.
You can clearly see this "week-end" discrepancy if you zoom on the Alphas: for instance (in additio to the first-day launch discrepancy), especially during the holiday week-end (Memorial Day: 23-24-25 May)
https://www.quantconnect.com/alpha/1c3f73cf79402b99741f85857Derek Melchin
Hi Laurent,
Thank you for the suggestion. The team will consider this change to forward-fill the OOS values during non-trading days.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jay Ryu
Hi, I recently deployed a new algorithm live, but I don't see oos values. Are the values backfilled periodically?
Derek Melchin
Hi Jay,
OOS values are updated at the end of each trading day.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!