Long Processing Time

There could be several reasons why your algorithm is running slowly. Here are a few possibilities based on the information provided in the extracts:

1. Data Resolution: If you're using a high-resolution data (like tick or second), it could slow down your algorithm because there are many more data points to process each day. As mentioned in the forum discussion, it's recommended not to use tick resolution unless necessary due to its slowness.

2. Accessing Data: If you're frequently accessing C# objects in your Python algorithm, it could slow down your algorithm. As mentioned in the documentation, it's faster to access them once and save a reference if you need to access them again. Here's an example:

# Do this:
security_holding = self.Portfolio[self.symbol]
avg_price = security_holding.AveragePrice
quantity = security_holding.Quantity

# Avoid this:
avg_price = self.Portfolio[self.symbol].AveragePrice
quantity = self.Portfolio[self.symbol].Quantity

1. Universe Selection: If your universe selection function is complex, it could slow down your algorithm. As mentioned in the forum discussion, QuantConnect calculates everything "online" and uses identical code for backtesting and live trading. If you're doing complex calculations in your universe selection function, it could slow down your algorithm.

As for your question about getting a “snapshot” of market data, QuantConnect does not currently support this feature. The data is streamed in real-time for live trading. However, they have started brainstorming for pre-synchronizing all the data which could potentially speed up the algorithm.

If none of these suggestions help, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.

Thank you for the above advice, would programming in C++ rather than Python make a substantial difference in processing time?  

Given that the calculations are not that intense, and there are no complex calculations in the universe selection function, it seems that the issue is more about the live data access / feed

QuantConnect's LEAN engine currently supports C# and Python programming languages. While C# is faster than Python, the difference in execution speed is more noticeable in backtesting rather than live trading. As mentioned in the forum discussion, the execution time of Interactive Brokers (IB) is far greater than any language difference. The only real speed difference you'll see is in backtesting where C# is 2-3x faster. This might only matter for options/futures algorithms.

If the calculations are not intense and there are no complex calculations in the universe selection function, the issue might indeed be related to the live data access/feed. However, it's important to note that QuantConnect streams data in real-time for live trading and does not currently support a “snapshot” of market data.

Switching to C# might bring some performance improvements, but it's unlikely to be a substantial difference in the context of live trading. If the performance issue persists, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.

Thank you again for your help

Is it reasonable to expect a scanner algorithm to process within say 2-3 seconds?

e.g Fine universe selection being to refine the market cap (eg between 3 Billion and 6 Billion), resulting in 500 stocks, and then to consider the last few seconds trade volume to the typical volume over the previous hour and accompany this by a drop in price?

Or, is there a structural data access reason why this might take an extended time?

What is an indicative timeframe for a typical scanning algorithm?

The time it takes for a scanner algorithm to process can vary greatly depending on several factors such as the complexity of the calculations, the number of stocks being processed, and the resolution of the data being used.

In the case of your example, where you're using fine universe selection to refine the market cap and then considering the last few seconds trade volume compared to the typical volume over the previous hour, it's important to note a few things:

1. Universe Selection: As mentioned in the documentation, both coarse and fine universe selection run on a daily basis. The live data for coarse and fine universe selection arrives at 7 AM Eastern Time (ET), so coarse and fine universe selection runs for live algorithms between 7 and 8 AM ET. This timing allows you to place trades before the market opens.

2. Data Resolution: If you're using high-resolution data (like second or minute), it could slow down your algorithm because there are many more data points to process each day.

3. Memory Usage: Each asset in the universe consumes approximately 5MB of RAM, so you may quickly run out of memory if your universe filter selects many assets. To keep your algorithm fast and efficient, only subscribe to the assets you need.

Given these factors, it might be challenging to process such a scanner within 2-3 seconds, especially if you're using high-resolution data and processing a large number of stocks. However, without seeing the specific implementation of your algorithm, it's hard to provide an exact timeframe.

If you're experiencing performance issues, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.