Share New Research 
Hi,
I have a custom data source for BTCUSDT from binance in minute resolution that I have used successfully in the past via .AddData(). However for ML model training, when I try to get consolidated history, the pandas Dataframe returned by self.History() seems wrong.
Data setup:
self.ticker = "BTCUSDT"
self.symbols = [self.AddData(type=BinanceTradeBarData, ticker=self.ticker,resolution=Resolution.Minute).Symbol]I call ML model training function via `self.Train(self.Training) within Initialize():
def Training(self):
...
# get historical data
# hist_data:pd.DataFrame = self.History(self.symbols, 20) # works + returns dataframe with shape (20,5) in minute resolution
"""
close ... volume
symbol time ...
BTCUSDT.BinanceTradeBarData 2S 2022-03-14 23:41:00 39625.53 ... 16.35635
2022-03-14 23:42:00 39638.02 ... 16.54291
2022-03-14 23:43:00 39645.74 ... 12.47931
2022-03-14 23:44:00 39648.04 ... 11.78058
2022-03-14 23:45:00 39545.50 ... 53.39393
2022-03-14 23:46:00 39555.17 ... 30.92626
"""
# hist_data:pd.DataFrame = self.History(self.symbols, 20, Resolution.Hour) # works + returns dataframe with shape (1200,5) with duplicate
"""
close ... volume
symbol time ...
BTCUSDT.BinanceTradeBarData 2S 2022-03-14 04:01:00 38177.89 ... 15.30639
2022-03-14 04:01:00 38150.91 ... 11.90438
2022-03-14 04:01:00 38135.00 ... 17.80335
2022-03-14 04:01:00 38185.00 ... 9.66609
2022-03-14 04:01:00 38209.61 ... 36.55811
... ... ... ...
2022-03-14 23:01:00 39689.81 ... 48.53675
2022-03-14 23:01:00 39687.24 ... 21.09104
2022-03-14 23:01:00 39691.64 ... 80.96595
"""
hist_data:pd.DataFrame = self.History(self.symbols, timedelta(hours=5), Resolution.Hour) # works + returns dataframe with shape (300,5)
"""
close ... volume
symbol time ...
BTCUSDT.BinanceTradeBarData 2S 2022-03-14 19:01:00 38700.00 ... 30.80545
2022-03-14 19:01:00 38711.01 ... 23.43854
2022-03-14 19:01:00 38747.86 ... 61.53202
2022-03-14 19:01:00 38706.69 ... 8.87105
2022-03-14 19:01:00 38699.11 ... 25.44933
... ... ... ...
2022-03-14 23:01:00 39689.81 ... 48.53675
2022-03-14 23:01:00 39687.24 ... 21.09104
2022-03-14 23:01:00 39691.64 ... 80.96595
2022-03-14 23:01:00 39674.91 ... 22.56632
"""
# hist_data = self.History(symbols = self.symbols, periods = 10, resolution = Resolution.Hour) # doesn't work: returns MemoizingEnumerable[Slice]
# hist_data = self.History(symbols = self.symbols, periods = 10) # doesnt work: returns MemoizingEnumerable[Slice]
# hist_data = self.History(symbols = self.symbols, span=timedelta(hours=5)) # doesnt work: returns MemoizingEnumerable[Slice]
# hist_data = self.History(symbols = self.symbols, start=datetime(2022,3,15), end=datetime(2022,3,20)) # doesnt work: Count of resolved generics 0 does not match method generic count 1.
As you can see none of the returned dataframes are in Hourly consolidated pattern.
Trial #1: returns data in how I added my dataset (=minute resolution)
Trial #2: returns data with duplicate `time`s.
Main question:
To get consolidated History data, should I retrieve via History with resolution not set and do pandas.groupby() etc to convert it to consolidated bars? Or is there a bug within custom data sources?
References checked
Any help is appreciated, thanks!
Fred Painchaud
Hi Notta,
Check out the documentation for History (https://www.quantconnect.com/docs/algorithm-reference/historical-data). It does not consolidate data. The period param is not for consolidation, it is to tell which period of time in the past for which you want data…
If you want to consolidate your custom data, you need to use consolidators. You can use them exactly like for usual data - you can check out the doc about consolidators (https://www.quantconnect.com/docs/algorithm-reference/consolidating-data).
Fred
Nottakumasato
Fred Painchaud How would I use consolidators after a self.History() call exactly? The documentation has no example of using a consolidator using the pandas DataFrame returned from the self.History() call.
Fred Painchaud
Hi Notta,
In a nutshell, consolidators have an Update() method which you can use to send them bars to consolidate.
Here's an example:
Fred
Nottakumasato
Fred Painchaud I used the consolidation_handler to append to a list so that I could convert it to a pandas DataFrame and use it as the training data for the ML model. Not sure if that's the best way to keep the bars created by the consolidator within self.Training() but works nevertheless.
Thank you for all the help!
Nottakumasato
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!