Hello
We have a scanner algorithm (in python), and its painfully slow. It takes 2-3 minutes to do what Excel does in under a second.
It processes / scans 500 stocks. We only have 1 small live processor, but I can't believe that in itself is the issue. There's not much calculation involved.
What could be the cause of delay? Is it the live data updating, if so can we get a “snapshot” of market data and then reload data after each iteration?
Please let me know of any advice, thanks
Mia Alissi
There could be several reasons why your algorithm is running slowly. Here are a few possibilities based on the information provided in the extracts:
Data Resolution: If you're using a high-resolution data (like tick or second), it could slow down your algorithm because there are many more data points to process each day. As mentioned in the forum discussion, it's recommended not to use tick resolution unless necessary due to its slowness.
Accessing Data: If you're frequently accessing C# objects in your Python algorithm, it could slow down your algorithm. As mentioned in the documentation, it's faster to access them once and save a reference if you need to access them again. Here's an example:
As for your question about getting a “snapshot” of market data, QuantConnect does not currently support this feature. The data is streamed in real-time for live trading. However, they have started brainstorming for pre-synchronizing all the data which could potentially speed up the algorithm.
If none of these suggestions help, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
John Hookway
Thank you for the above advice, would programming in C++ rather than Python make a substantial difference in processing time?
Given that the calculations are not that intense, and there are no complex calculations in the universe selection function, it seems that the issue is more about the live data access / feed
Mia Alissi
QuantConnect's LEAN engine currently supports C# and Python programming languages. While C# is faster than Python, the difference in execution speed is more noticeable in backtesting rather than live trading. As mentioned in the forum discussion, the execution time of Interactive Brokers (IB) is far greater than any language difference. The only real speed difference you'll see is in backtesting where C# is 2-3x faster. This might only matter for options/futures algorithms.
If the calculations are not intense and there are no complex calculations in the universe selection function, the issue might indeed be related to the live data access/feed. However, it's important to note that QuantConnect streams data in real-time for live trading and does not currently support a “snapshot” of market data.
Switching to C# might bring some performance improvements, but it's unlikely to be a substantial difference in the context of live trading. If the performance issue persists, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
John Hookway
Thank you again for your help
Is it reasonable to expect a scanner algorithm to process within say 2-3 seconds?
e.g Fine universe selection being to refine the market cap (eg between 3 Billion and 6 Billion), resulting in 500 stocks, and then to consider the last few seconds trade volume to the typical volume over the previous hour and accompany this by a drop in price?
Or, is there a structural data access reason why this might take an extended time?
What is an indicative timeframe for a typical scanning algorithm?
Mia Alissi
The time it takes for a scanner algorithm to process can vary greatly depending on several factors such as the complexity of the calculations, the number of stocks being processed, and the resolution of the data being used.
In the case of your example, where you're using fine universe selection to refine the market cap and then considering the last few seconds trade volume compared to the typical volume over the previous hour, it's important to note a few things:
Universe Selection: As mentioned in the documentation, both coarse and fine universe selection run on a daily basis. The live data for coarse and fine universe selection arrives at 7 AM Eastern Time (ET), so coarse and fine universe selection runs for live algorithms between 7 and 8 AM ET. This timing allows you to place trades before the market opens.
Data Resolution: If you're using high-resolution data (like second or minute), it could slow down your algorithm because there are many more data points to process each day.
Memory Usage: Each asset in the universe consumes approximately 5MB of RAM, so you may quickly run out of memory if your universe filter selects many assets. To keep your algorithm fast and efficient, only subscribe to the assets you need.
Given these factors, it might be challenging to process such a scanner within 2-3 seconds, especially if you're using high-resolution data and processing a large number of stocks. However, without seeing the specific implementation of your algorithm, it's hard to provide an exact timeframe.
If you're experiencing performance issues, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
John Hookway
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!