Research notebook being very slow...

Based on the information provided in the extracts, there could be several reasons why your research notebook is running slow. Here are a few potential causes and solutions:

1. Python-C# Interactions: As mentioned in the forum discussion, the interaction between Python and C# could affect the speed of your notebook. For instance, a single line of code (especially if it involves pandas) could significantly slow down your notebook. It's recommended to understand what types of operations are more efficient and adjust your code accordingly.

2. Data Access: As discussed in the forum discussion, if your algorithm is trying to access a large amount of data, it could slow down the process. QuantConnect's LEAN engine streams data in real-time for live trading, which could potentially slow down your algorithm if you're trying to access a large amount of data at once.

3. Algorithm Complexity: If your algorithm is complex or involves computationally-intensive tasks, it could slow down your notebook. As suggested in the forum discussion, you could consider reducing the size of your model or using less data to train your models more periodically.

4. Hardware Limitations: If you're using a free account, you might be on shared hardware which could affect the speed of your notebook. As mentioned in the forum discussion, LEAN does support a few thousand companies, but loading a large amount of data could cause issues on shared hardware.

If none of these suggestions help, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.

Sometimes it even just get stucked there without executing. After waiting for maybe 10 mins when the engine decided to execute, it takes another 15 min - 20 min or so to load something really simple… The last block just finished being executed after 21 min…

Based on the information provided in the extracts, it seems like you're experiencing performance issues with your research notebook. Here are a few potential causes and solutions:

1. System Updates: As mentioned in the forum discussion, QuantConnect's engineers have updated LEAN and the system, which has sped things up a bit. They're also making some front-end updates which they expect to result in performance improvements. You can track the progress of these changes and any updates on GitHub.

2. Execution Time: In the forum discussion, a user mentioned that the execution (backtest) seems to take forever (minutes). The deployment takes ~15s or more. This could be due to the load on the backtesting nodes at the time you submitted the backtest. Live servers are generally a bit faster than backtest nodes.

3. Data Points: In the same forum discussion, a user mentioned that they were trying to log strike deltas for 2018 and it was taking a long time. This could be due to the large number of data points being processed.

If none of these suggestions help, it would be beneficial to share your algorithm with the QuantConnect support team so they can help identify the root cause of the slowness.

Hi Brian, please post a full research notebook demonstrating the issue, so we can assist. 

Thanks Jared for the prompt response. Please see attached one of the research notebooks that takes a long time. In general I think it occasionally happens in all the notebooks. Attached is just an example. 

Sorry. Now attaching the project and one backtest. Did not know we need to have a backtest attached to post the notebook.

Hi Brian,

I thoroughly enjoyed reviewing your extensive code! Excellent job on your work.
I did not encounter any instances of sluggish notebook performance in the provided research notebooks.

I comprehend that the specific code snippet you mentioned initiates with the Consolidator_2 method, which invokes the History method with Resolution set to Second and Frequency set to 10. This operation should not be causing the observed delay. The subsequent functions involving slicing and indexing are straightforward and complete within a fraction of seconds.

I would suggest examining aspects on your end and, if possible, debugging any elements in the execution of your code or the overall system that might be contributing to the observed slowness.

Let us know if you need any assistance!

Cheers,
Ashutosh

Attaching some screenshots for reference: