Long backtests ending or freezing?

Hey Justin E, it’s quite difficult to know the exact cause of this behavior(because you didn’t provide the code). Following points could be the reason:

• Your node is to „weak“ for the things you have to calculate 
• your pc goes into power saving mode, you would have to deactivate it (speaking of personal experience)
• the code might have some loop errors / general errors
• if the code is longer than 10 minutes in a loop the backtest stops
• the backtest is finished but because of a bug its not displayed→ close project and open project again

You could try out a better node, if nothing helps you could set up a screen recording and look for the cause there. 

Hope it helps ;)

Hi Nico Xenox 

I wish I could provide the code but my strategy would be on full display.

I'm confused how a node can be weak or bad? How do I select or upgrade to a ‘stronger’ one?

Also, how does my PC affect the backtest if it's running in the cloud? I should be able to close the browser tab right?

Justin

As a follow up, I'm probably hitting the 12 hour limit.

My RAM is around 25% usage of max, I've debugged all loop errors as the code is stable, and I've turned off sleep mode a long time ago. So the only thing I can think of is the 12 hour limit. 

Would upgrading to a B4-12 result in faster backtests? 

Justin

Hey Justin E, 

did you try a better node? Are both CPU and RAM in your backtest at the max or not? Also is the reason for this slow behavior a lot of history requests or do you have an universe of symbols? If you try to minimize each factor to a minimum you might reeceive your answer. If it is the number of symbols then I would personally upgrade your node. If it is a history request with a set of symbols try to lower the timespan or to do a lower resolution.(Hour, daily). Another option which could work for the history request but might involve a little more work would be importing the history data via objectstore / obj. 

Hope it helps ;)

Hi Nico Xenox 

This was helpful because I am using 700 symbols after the Fine Filter but only about 50-100 symbols are traded daily with new data coming in every 5 seconds. So, to your point, would upgrading my B2-8 node to a B4-12 increase speed? 

I do see my CPU is past maximum, usually running around 150%. The screenshot below is from a backtest I'm running right now, notice the CPU past max but RAM seems to be fine.

And this algo doesn't use any history requests. 

Justin

Hey Justin E,

nice that we found the problem. If you want to keep all the symbols you mentioned you probably should try it out. Jared told me the payment form is prorated so you will be charged failry(you will notice if you cancel how it works) if you want to go back to the old one. Let me know if by upgrading your node it works 😊

Will do Nico Xenox, I'll upgrade tonight and re-run these backtests. I'll close this discussion after I post the results here to this thread. 

Thanks again!

Justin

Hey Justin, 700 securities represent roughly 20% of the Morning Star dataset / 2TB of data! It's likely running out of RAM or terminating after the 12 hours. We can review it for optimization if you want to share it here or send in a support ticket. There are leaks in some python packages that complicate things. 

The algorithm would be fastest in C# and use the full allocated CPU (200% or 400% etc.). The slowest part of python algorithms is the python algorithm thread. 

Hi Jared Broad and Nico Xenox,

I attached a backtest, I would be grateful for a little diagnostic on the algo. 

The 700 tickers are set to be rebalanced every 35 days, and only ~50 tickers are actually getting the 5 seconds of data during trading hours after I pass the 700 through a tertiary filter once at the beginning of each trading day.

And I attached a backtest, it's only of 2 days to make sure it started correctly as I removed the sensitive decision making code and replaced it with something closely resembling it. These backtests have been 1.5 months in length. In the screenshots below they were 6-1-2022 to 7-15-2022.

Regarding the node upgrade results: I ran the same backtest, one with a B2-8 node and the other with a B4-12. The results were almost identical, see the screenshots below for the CPU and RAM usage. Total up time was similar, around 4.5 hours. I'm not sure why the results are the same even down to CPU utilization? I would have thought the CPU utilization would have dropped in half if the CPU cores doubled. I'm no computer science expert so I could be misunderstanding CPU and RAM altogether. 

B2-8 node

B4-12 upgrade

Justin

Hey Justin E,

As Jared already mentioned you use quite a lot of tickers in your algo. This makes the algo extremly slow. For one day using 200 stocks it needs 1:14 minutes. 11 days with the same number of stocks need nearly 10 minutes. I tested it with the best node you can currently get. Doing backtests with that amount of symbols will be quite challenging. Meaning that a whole year will probably take 7 or more hours to complete depending on what node you have. Also I'm not sure that you will find a solution that will include all 650 symbols to make all work in a matter of time that the backtest will be able to finish. 

Currently I would advice you to lower the number of stocks and do the backtesting on a short timeframe…

HI Nico Xenox,

Thanks for taking a look. I was able to reduce the ticker universe by 25% which has improved run time. Philosophically, I'm trying to keep the universe large enough to prevent bias related to overfitting while creating diversity of trade opportunities. It also doesn't help that only a fraction of the total universe end up becoming eligible trades, hence the large size in the first place. 

I appreciate the support on this!

Justin