Most Efficient Way Of Creating A DataFrame of Historial Daily Closes In Algo

Hi Jonathan,

I would recommend that you use Rolling Windows for this.

They will initially grab the historical data for you but then simply update the dataframe with the new values every day as the data comes into your algorithm.

This should save you a lot of time when backtesting.

Let me know if you have further questions, thanks!

The Rolling Windows part is just a way of storing the data, correct? Does it really make it different performance-wise if I stored the data in a dictionary vs in a RollingWindow structure?

It's only daily data, so it's not that much.

Hi,

If you use the Rolling Windows you avoid calling a lot of data everyday, which is what slows down your algorithm by a lot.

Every history call that you are making you are calling that entire data again.

So it is not so much about the storing of data but the calling of the data.

Generally it is advised by QuantConnect to use History Calls sparingly

Let me know if you have any further questions, thanks!

To be clear, I'm not calling self.History in the code example below “So I tried this approach:”

I'm writing data within a consolidation handler.

And writing data to a RollingWindow or writing data to a dictionary shouldn't make difference performance-wise, right?

Hi,

I' assuming that your consolidation is grabbing each price over and over again instead of saving them and simply adding the last value and removing the oldest.

A Rolling Window will perform one history call at the start of your algorithm and then pop the oldest value and append the newest each day while keeping all of the other values untouched in the Rolling Window.

Please try out the Rolling Window and get back to me with the results or any issues that you face, thanks!

This is what I'm doing:

def Initialize(self):
   ...
   for symbol in symbols:
       self.Consolidate(
           symbol  = symbol,
           period  = Resolution.Daily,
           handler = self.process_daily_bar)

   self.history = {}

def process_daily_bar(conosolidated):   
   symbol = str(consolidated.Symbol)
   date  = consolidated.EndTime.date()
   self.history[symbol, date] = consolidated.Close

I emailed support, and they have confirmed that RollingWindow is not necessarily more efficient. It really depends on how often you plan on calling self.History.

Using RollingWindow in itself is not more efficient than any other data structure.

Hi Jonathan,

Mak's point was that calling History over each day for 100 assets is going to cost a lot performance wise.

He meant that instead of calling History, you could store the day data as it goes into a RollingWindow.

But with only a fraction of Initialize and one function, it is difficult to further help spotting the performance hog.

Cheers!

Fred