Share New Research 
A python algorithm that selects 400 securities at minute resolution, with empty OnData does ~250K points per second.
Adding a consolidator for each security and registering it with the SubscriptionManager reduces it to ~50K points per second.
If we instead manually call Consolidator's Update() method in OnData(), it improves to ~125K points per second.
All reproducible in attached backtest.
Questions:
- Why does using the SubscriptionManager cause such a drastic slowdown?
- Is there anything that can be done to improve performance of TradeBarConsolidator? The aggregation logic does not appear overly complex so it is surprising that it reduces throughput by 50%.
Arthur Asenheimer
Hi Karthik,Â
I've noticed this as well. Adding user-defined consolidators results in a significant slowdown, at least for Python algorithms.Â
It has also been discussed here before. There it was justified with the fact that consolidators bring additional workload. However, this cannot be the reason as I will demonstrate below.Â
Here is the result when we register an indicator by using RegisterIndicator():
Here is the result when we register an indicator by using an user-defined consolidator:
Both backtests had the same workload (number of processed data points is identical), but the second version with an user-defined consolidator is ~70% slower.Â
My guess is that with RegisterIndicator() the indicator updates are executed with C# speed, whereas with a user-defined consolidator the indicator/consolidator updates are executed inside a Py.GIL block and thus with Python speed.Â
This is also confirmed by the fact that for C# both versions run with almost identical speed (see next post). This shows that the problem is exclusively for Python.Â
I've created a GitHub issue in the hope that the QC software engineering team can give us more information on this and investigate the issue further.Â
Â
Arthur Asenheimer
Here the same algorithm written in C#. Just set selectUserDefinedConsolidator = true/false to test both versions. Note that the performance is almost the same here.Â
Using RegisterIndicator:
Â
Using user-defined consolidator:
Jared Broad
>whereas with a user-defined consolidator the indicator/consolidator updates are executed inside a Py.GIL block and thus with Python speed.Â
You're correct Arthur Asenheimer; given it's a C# object it shouldn't be crossing the C-Python boundary. We'll look into why it's slow and create the right object. We do expect user-defined python ones to be slower though since to do the consolidation it needs to cross the boundary.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Karthik Kailash
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!