Best Practice for Getting Historical Custom-Consolidated Data?

Hi LE,

When you use the automatic warm up feature, it uses the resolution you are registered to, which makes sense of course.

But when you manually call History, you can query the resolution you want. Thus, if you want to warm up an indicator with 21 days of some daily resolution, you would then get daily bars instead of second bars and it would take a whole less time to get the result. I would then not bother about the weekly resolution if it is to speed things up. Normally, your weekly consolidator should be able to take days and make weeks with them…

So calling manually History would be:

df = self.History(eurusd, timedelta(21), Resolution.Daily)

That will return the daily bars for the last 21 days (take note they are calendar days).

Fred

Thank you Fred!

I need the weekly consolidator to get the OHLC data for the week, a weekly quotebar - I don't mind using daily data. How do I get the daily data from the history call into my custom consolidator and produce weeks, and will it automatically feed the registered indicator? Currently I have:

###(in Initialize)

	#Set Up
	w = QuoteBarConsolidator(self.CustomWeeklyConsolidator)
	w.DataConsolidated += self.OnDataCustomWeekly
	self.SubscriptionManager.AddConsolidator(self.pair, w)

	#Register Indicator
	self.customIndicator = self.indicators.CustomIndicator('weeklyIndicator', None, 1)
	self.RegisterIndicator(self.pair, self.customIndicator, weeklyconsolidator)
	
###(elsewhere)

def CustomWeeklyConsolidator(self, dt: datetime) -> CalendarInfo:
        period = timedelta(7)

        dt = dt.replace(hour=17, minute=0, second=0, microsecond=0)
        delta = 1+dt.weekday()
        if delta > 6:
            delta = 0
        start = dt-timedelta(delta)

        return CalendarInfo(start, period)

	

Hey LE,

You need to keep the consolidator unregistered as this will make it auto-update with your data sub. So:

w = QuoteBarConsolidator(self.CustomWeeklyConsolidator)
w.DataConsolidated += self.OnDataCustomWeekly
#self.SubscriptionManager.AddConsolidator(self.pair, w) #remove

So then your conso knows how to consolidate and also knows where to send consolidated bars. But it does not receive any data to consolidate. I will go back to that later.

This is ok:

self.customIndicator = self.indicators.CustomIndicator('weeklyIndicator', None, 1)
self.RegisterIndicator(self.pair, self.customIndicator, weeklyconsolidator)

Or at least I am guessing that you defined a CustomIndicator class and you thus simply create an instance and register it to get weekly consolidated bar in the indie.

From the name of the variable, “weeklyconsolidator", I am assuming you are passing a consolidator to RegisterIndicator. Note that it must not be registered. So it looks like it was not the same as “w” above, which is good since “w” was registered. Note that if you double register consolidators like that, the last time I checked, LEAN will not warn you but you will get wrong data, especially if you use volume. Anyway, I am just saying as I go…

Now, back to feeding “w” with data.

Since you are subscribed to second data, you do not want to use OnData slices to update your conso. This would be equivalent to automatic warmup.

So you want to use the result of a History call. You should make that call somewhere it makes sense to warmup your indie, normally, after creating it, since then, you usually know for which symbol you are creating your indie. And it would look like that (doing my best to infer your own variables):

history = self.History(self.pair, timedelta(21), Resolution.Daily)
for index, row in history.iterrows():
    quote_bar = QuoteBar(index[1] - timedelta(1), self.pair, Bar(row.bidopen, row.bidhigh, row.bidlow, row.bidclose), row.bidsize, Bar(row.askopen, row.askhigh, row.asklow, row.askclose), row.asksize, timedelta(1))
    w.Update(quote_bar)

Fred

My mistake when formatting my post, “weeklyconsolidator” was indeed supposed to be “w” in the RegisterIndicator call. Does that mean it would be “double registered” as you say? Does the SubscriptionManager call interfere with the RegisterIndicator call somehow? I have not had any issues yet (not using volume, just quotebars/prices)

Ah, thanks to your code I see now how to update “w” manually with the incoming history data, it's quite easy.

However, I do need “w” to keep updating automatically (every week) as the algorithm runs after warmup. Will it do that without the subscription manager call?

I'm essentially just trying to reduce the amount of data for the warmup, after that I'm fine with the indicator auto-updating.

Thanks again Fred, this is incredibly helpful!

I can't say for sure without full code (not saying that to get the full code) but it seems like you were double registered, yes.

Yes, RegisterIndicator calls SubscriptionManager.AddConsolidator, so yes, they “interfere”. Or… what they both do intersect to some extent. Check

https://lean-api-docs.netlify.app/QCAlgorithm_8Indicators_8cs_source.html#l02499

and

https://lean-api-docs.netlify.app/QCAlgorithm_8Indicators_8cs_source.html#l02573

To keep your weekly conso happy with data throughout weeks and not only for warm up, you can then ALSO manually feed it with data in OnData. This time, no need to call History:

def OnData(self, slice: Slice) -> None:
    quote_bar = slice.QuoteBars[self.pair]
    w.Update(quote_bar)

I am just getting the quotebar in the slice received. Note that registering consos basically simply do that right before calling OnData, in the background.

Another option would be to first NOT register the conso to warm it up manually and THEN register it so it will begin to receive the next second quotebars. Make sure you use the most recent version (if Cloud, you are) of LEAN since there was a commit recently breaking that use case. But it's back in business.

Don't worry LE…

Okay, so I don't lose anything if I just get remove the SubscriptionManager.AddConsolidator call? What's the intended use case for SubscriptionManager?

I think warming it up manually and then registering it is easier for me than manually updating through OnData. Thanks for the suggestion! I am on the Cloud version.

Well, SubscriptionManager.AddConsolidator adds a consolidator to the first subscription that is valid for the consolidator. See

https://lean-api-docs.netlify.app/SubscriptionManager_8cs_source.html#l00149

Being valid has been implemented as 1) same symbol and 2) assignable input types between conso and symbol's data type.

SubscriptionManager itself is something else, it manages subscriptions! 😊 No kidding, it is where all data subscriptions end up and can be manipulated.

RegisterIndicator is a helper method to 1) register the indicator with the subscriptions manager and 2) connect the passed consolidator to the indie so the consolidated bars are sent to update the indie. So it says “I want data from this symbol feeding this conso and consolidated bars from this conso feeding this indie”.

Thank you for the information! I will try to keep your advice in mind. 

For some reason, I am getting this error with the history call, do you know why? In the docs I don't see anywhere for the quotebar parameter?

#relevant code
self.pair = "US30USD"
self.AddCfd(self.pair, Resolution.Second, Market.Oanda)
history = self.History(self.pair, timedelta(21), Resolution.Daily)

#error

"Calling History<TradeBar> method on a Forex or CFD security will return an empty result. Please use the generic version with QuoteBar type parameter."

Oh yeah… try to use History[QuoteBar](self.pair, timedelta(21), Resolution.Daily).

Thanks, I changed it to 

history = self.History[QuoteBar](self.pair, timedelta(21), Resolution.Daily)

and it seems to have fixed the tradebar issue, and made it through that line. However on the next line it seems like it's returning an empty dataframe? 

#code

history = self.History[QuoteBar](self.pair, timedelta(21), Resolution.Daily)
for index, row in history.iterrows():
	...

#Error

"During the algorithm initialization, the following exception has occurred: 'MemoizingEnumerable[QuoteBar]' object has no attribute 'iterrows',  at Initialize,    for index in main.py: line 238, 'MemoizingEnumerable[QuoteBar]' object has no attribute 'iterrows',ApiConnection.TryRequest(backtest/status/update): Error: The operation has timed out."

Crap…

The issue is that you are not receiving a Python object (DataFrame) but a C# object with the call. So of course, iterrows is a DataFrame method… For some reason, it's not processed by Python.NET to convert the MemoizingEnumerable (C#) into a DataFrame (Python)… Most likely it is simply because those C# method flavors are not implemented into the Python stub for History calls.

The other option to try is to go with the symbol collection flavor of History instead of the generic flavor (the one with the [] above). So:

self.History([self.pair], timedelta(21), Resolution.Daily)

If this is still not working for the same reason, then it is most likely the flavors with timedeltas (spans in the docs) that are not properly processed or implemented… I already reported that in another forum post but it will be looked at later.

So then, you could go back to the version with a number of bars:

self.History([self.pair], 21, Resolution.Daily)

Just like you were using.

The difference should be that timedelta(21) gives you 21 calendar days back. So holidays and closed days count. 21 would give you 21 bars, so then only over trading days in theory - that's the intention. At Daily resolution, it works for sure because an asset must be very illiquid not to trade for an entire open day. But for seconds, if you count say 1000 seconds back from “now” during open hours, it does not mean that each of those “open” seconds actually have a second bar (there can be no trades over one second in particular, even though the market is open for that asset). So for short resolution like that, History will not return the number of bars specified but less (as in self.History(self.spy, 1000, Resolution.Second - it will most likely not return 1000 bars, the more bars you ask for (at second), the less you will receive compared to your request since the more untraded periods you will encounter…).

I think you can convert result to pandas dataframe with 

self.PandasConverter.GetDataFrame(history)

Thanks for the info Fred. Because you mentioned “symbol” I think I solved it: the “self.pair” variable is just text. In the docs they use a symbol object. I changed it to this and now I make it into the for loop!

self.pair = "US30USD"
mypair = self.AddCfd(self.pair, Resolution.Second, Market.Oanda).Symbol

#history call

history = self.History(mypair, timedelta(21), Resolution.Daily)
for index, row in history.iterrows():
    quote_bar = QuoteBar(index[1] - timedelta(1), self.pair, Bar(row.bidopen, row.bidhigh, row.bidlow, row.bidclose), Bar(row.askopen, row.askhigh, row.asklow, row.askclose), timedelta(1))
    w.Update(quote_bar)

#error
During the algorithm initialization, the following exception has occurred: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the 'datetime.timedelta'>) method. Please checkout the API documentation.
  at Initialize
    Bar(row.askopen in main.py: line 240
 No method matches given arguments for .ctor: (<class 'pandas._libs.tslibs.timestamps.Timestamp'>, <class 'str'>, <class 'QuantConnect.Data.Market.Bar'>, <class 'QuantConnect.Data.Market.Bar'>, <class 'datetime.timedelta'>)

However it starts throwing some other errors (first I had to remove bid/asksize since it's a quotebar) (now it doesn't like timedelta). Would it be easier to do it like in the docs:

self.pair = "US30USD"
mypair = self.AddCfd(self.pair, Resolution.Second, Market.Oanda).Symbol

#history call

history = self.History(mypair, timedelta(21), Resolution.Daily)
if not history.empty:
	my_quotebars = history.loc[mypair] #self.pair should work here too
for index, val in my_quotebars:
	w.Update[my_quotebars[index]]

That's the idea of it, but I'm not sure those are actual quotebars I can pass. What do you think?

Thank you too for the additional option, Yuri!

Hey Yuri,

You are right. Thanks. Never spotted that converter stuff. From the signature, that method would do it:

PyObject GetDataFrame< T > (IEnumerable< T > data)

Seeing that conversion stuff makes me realize new stuff about how pythonnet works… good.

Hey LE,

Sorry for the string thing. Of course, always use Symbol objects all around. Make it a habit as soon as possible. In fact, one good way is not to store tickers at all like “SPY”. Use the string directly in the first few calls you need to make and then only keep Symbols… anyway.

Sorry but I don't see your code in the docs you referred. The loc[mypair] will work and will simply get the first indexing level out (Symbol). You'll then only have time as an index. Fine but not necessary, I mean, fine, you know, whatever :).

You can try it but I really don't think

for index, val in my_quotebars:

will work as a DataFrame is not iterable per say in Python. You need to call iter*() methods. But well, never bad to try.

And if

w.Update[my_quotebars[index]]

works, then I should really really have another look at Pandas DataFrame's implementation :). What is sure is that a df will not hold QuoteBars directly. You need to create QuoteBars from the data in the df… Try not to mess the times for the QuoteBar constructor. The first param is the start time but in the df you get the end time. Moreover, the default value for the period, last param, is 1 minute so if you omit that param, you tell LEAN that your quotebars are for 1m, not 1d. It will most likely silently shift you entire strategy and except if you have a sharp eye for that kinda stuff, you won't even spot it when looking at plots and results…

In your reported error, the mismatched signature is because you changed the call. There is no such constructor with the params you are now giving.

If you had an error with asksize and bidsize it is because it was not in the data. So you should use 0.0 instead. Moreover, you are still using self.pair which is a string, not a Symbol, so again, the call would not match. There is no constructor taking a string and using the cache instead of taking a Symbol.

So:

#self.pair = "US30USD" #get rid of that
self.pair = self.AddCfd("US30USD", Resolution.Second, Market.Oanda).Symbol

#history call

history = self.History(self.pair, timedelta(21), Resolution.Daily)
for index, row in history.iterrows():
    quote_bar = QuoteBar(index[1] - timedelta(1), self.pair, Bar(row.bidopen, row.bidhigh, row.bidlow, row.bidclose), 0.0, Bar(row.askopen, row.askhigh, row.asklow, row.askclose), 0.0, timedelta(1))
    w.Update(quote_bar)

We are back to the History call which returned a Memoizable…. but I wonder if it was because of self.pair being a string, not a Symbol. Now it is a Symbol. If you still get the Memoize error, you can use the Converter as Yuri pointed out, or you can go back to 21 instead of timedelta(21). As explained, if you want 21 daily bars, use 21.

Fred

For the tip about Symbol objects, I'll try to do that. I hope that all my references to the text variable can simply accept Symbol if I change the original reference. 

I was just referencing the “analyze results” section of the history docs. Right, I didn't think they would be quotebars. 

That history/quotebar loop seems to work and the code runs fine! However now my custom indicator isn't happy and not outputting during the manual warmup, I'll have to check that haha.

Thanks again, I appreciate all your time helping with this!

Bonus Question: How do I self.Log() or self.Debug() from a separate custom indicator class (in a different file)? Not sure how to do it without the algorithm functions. 

No worries LE.

To call Log or Debug, you need a reference to your QCAlgorithm, which is self in the QCAlgorithm methods, such as Initialize, OnData, any other you add, etc.

So add a param to the __init__ method of your class, such as:

class Foo:
	def __init__(self, algo, whatever):
		self.algo = algo
		self.whatever = whatever
		
	def hey(self):
		self.algo.Log("HEY!")
		self.algo.Debug("AAAAARRRRRRRRRRRRGGGGGGGGGGG!")
		self.algo.Debug(self.whatever)

And then when you create instances inside some QCAlgorithm method:

class MyAlgo(QCAlgorithm):
	def Initialize(self):
		...
		self.foo = Foo(self, "Whatever!")

Fred

Got it, thanks!

I also fixed the issue with my indicator not running, I had to RegisterIndicator() to “w” before the history call of course. 

However if I don't use exactly timedelta(21) and start the algorithm on a Sunday, then the output is off.  Does the history call / consolidator just add the first seven days it receives regardless of date? 

(Forex/Futures/CFD week starting Sunday 17:00 EST)

I've attached a test algo for some of the cases. You can see in the logs how it changes if you change to timedelta(24) or so. I tried using hours and had the same problem (exact hours work but else not).

For example, the first log for the week of June 19 after the “Weekly Hello” (Sunday) should be 06-20, as in attached backtest B. But it is 06-21, if you change to timedelta(24).

I tried changing the period in “CustomWeeklyConsolidator” from timedelta(7) to just 7 but that did not help. 

Hi LE,

I can see that.

The issue is from requesting day bars. In the DB, for that symbol, day bars are from 19h00 to 19h00. And of course, they are not secable. Since your weeks are from 17h00 to 17h00, the data between 17h00 and 19h00 are in the next week bar, but that 19h00 to 19h00 bar cannot be divided into 19h00-17h00 and 17h00-19h00.

Using Hour instead of Daily for the history request makes things ok. You still have less data than second, but then, hour bars can easily be attributed to the right weeks…

See attached.

You still use timedelta(24) or timedelta(21) because it really means timedelta(days=24) or timedelta(days=21). So you are asking for 24 or 21 days worth of hour bars…

Even though I did not do it, I would recommend you to use timedelta(days=21) instead of timedelta(21), say. It is more explicit and it documents the code. Note you could do timedelta(days=6, hours=4, minutes=20), say, to get the last 6 days, 4 hours and 20 minutes……. (and you can specify seconds, etc).

Fred

Thanks Fred. I suspected as much, that it would be hard to fit the daily data blocks into the consolidator. 

However, when I used hours, I ended up with the same issue if I did not use the exact amount of hours (504) making a clean 3 weeks. When I used 600 my indicator results were off again. 

I would like the flexibility to be able to start my algo on a Wednesday or so, or change the warmup amount (as long as it’s greater one week), and not have any issues if possible. (Intended indicator output is once weekly at the start on Sunday) (so I would have the most recent Sunday’s results in-algo) 

I will double check later today and if you’re sure then the problem might be somewhere else. Or I’ll try to provide an example. 

Good to know about timedelta default to days, and I will use the explicit definition! 

Hi LE,

I thought the attached backtest worked but I was wrong?

history = self.History(self.pair, timedelta(24), Resolution.Hour) #history call for three weeks

Again, you can mix and match timedelta(…) with Resolution.Hour. You're saying “I want THAT timespan with THAT resolution” with that call…

So you want 3 weeks. That's 21 days. I'd say you want:

history = self.History(self.pair, timedelta(days=21), Resolution.Hour) #history call for three weeks

Of course, if you start a Wednesday, you'll get 3 weeks from that Wednesday. But that's 3 weeks still.

Are you saying you want at least 3 full weeks? So say you start on June 29th, you want the data to start on June 8th or June 5th? If June 8th, that's already what you have. If June 5th, then yes, it needs a little tweak:

history = self.History(self.pair, timedelta(days=21 + ((self.Time.weekday() + 1) % 7)), Resolution.Hour) #history call for three weeks, starting on a Sunday

Fred