Share New Research 
I've created an algo that I'm quite happy with, but the problem is that when I'm trying to start my backtest. I have to limit my universe, which should be most stocks but at least 3,000 as the start takes forever to load and most often timeouts due to the 10 minutes limit. I've now had to cut it down to 500 stocks which is not really feasible as it'll only give me large cap universe.
So I have a couple of questions.
- How can I log so I know which part of my code takes long to run and as such work on improving those bits?
- Can you override the 10 minute cap somehow?
- Any feedback on my init as attached to this question would be useful. I have simplified it and don't really show what I do with the factors I load. Thinking I should collate a few of the for-loops to one for-loop. But not sure if that would make a huge difference, feels like a small change?
AK M
I cloned your algorithm and I see what you mean. It is very slow but it never timed out for me.
I'm not sure what the exact reason is, but I did notice that once the algorithm starts executing, it finishes relatively quickly. So your timeout happens when your algorithm initializes. It's also possible it might be a data issue because when I run your during more recent timeframes, it initializes and executes faster.
I think its very possible that the universe is too big, so some of these filtering criteria may be helpful to you.
I added criteria to your coarse selection function to narrow down what ends up in fine selection. I added some liquidity checks. I recommend keeping them because you're going to want to have liquid equities anyways. I see that you check that your equities are traded on NYSE and NASDAQ in your fine selection, so I also added a filter to make sure your market is USA. Make sure these filters actually do what you want before you incorporate them.
selected = [x for x in coarse if (x.HasFundamentalData) and (x.Market == "usa") and (float(x.Price) > 1) and (x.DollarVolume > 1000000) and (x.Volume > 10000) ]I also made sure your traded securites have been in the Universe for atleast 180 days.
self.UniverseSettings.MinimumTimeInUniverse = 180Quanter123
Thanks for the feedback! Will try those out and see if they help. Although I need them to be liquid I'm actually more interested in mid and small cap companies so liquidity isn't my biggest concern, which is why I want the large universe to start with. It feels a bit strange that the engine doesn't support a few thousand companies given that would mean for example that you couldn't replicate something like Russell 2000.
Jared Broad
In general LEAN does support a few thousand, but I think because you're on a free account which is on shared hardware it's having trouble loading the 3000 fundamental data sets (about 50GB) within the 10min timeout.
To speed up the process once the data is loaded you can also return "Universe.Unchanged" instead of returning the same ticker list. If unchanged it will prevent loading the data on the intermediate days.
We're going to keep working on speed but you can track our progress here:
https://www.quantconnect.com/performanceThe material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Quanter123
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!