I'm interested in using the History() method to ingest a block of data from Tiingo. I'm using SPY just as test because it's available from both QC data and Tiingo. History() works fine, if I use:
self.AddEquity(self.ticker, Resolution.Daily)
h = self.History([self.ticker], TimeSpan.FromDays(5), Resolution.Daily)
However, if I write:
self.AddData(TiingoDailyData, self.ticker, Resolution.Daily)
h = self.History([self.ticker], TimeSpan.FromDays(5), Resolution.Daily)
I get the following error:
at Initialize in main.py:line 31
Exception : cannot handle a non-unique multi-index!
I was unable to figure out from the documentation how to use History() with TiingoDailyData (or other custom data source). Perhaps there is another way to retrieve blocks of historical data?
GregasMaximus
Being pretty new at working with this code, I'm not so familiar with the internals. After some probing, it seems clear that the interface for Tiingo data does not support History(). In order to support algorithms that require historical lookback, I wrote the following class that agreggates price data from slices. Not beautiful or necessarily efficient, but it works. If you like it, feel free to improve it and post back here.
class DataAggregator:
'''
A class for aggregating one field from a slice from external data source
It collect a rolling window of n points
tickers: List of tickers to extract
field: String defining which field to extract
n: number of points to keep
'''
def __init__(self, tickers, field, n):
self.tickers = tickers
self.field = field
self.n = n
self.i = 0
self.data= pd.DataFrame(index=range(self.n), columns=self.tickers)
def agg(self, slice):
if (self.i < self.n):
for t in self.tickers:
self.data.iloc[self.i][t] = getattr(slice[t], self.field)
self.i += 1
else:
self.data = self.data.shift(-1,)
for t in self.tickers:
self.data.iloc[-1][t] = getattr(slice[t], self.field)
To use, put something like this in your Initialize() method:
self.tickers = ['SPY', 'TLT']
self.history = DataAggregator(self.tickers, 'AdjustedClose', 200)
Then, in OnData(), put:
self.history.agg(slice)
I did not try to replicate the full functionality of History(), but this simple hack should serve for most simple use cases.
Greg
GregasMaximus
Quick update: I cleaned this up a bit. I looked at using RollingWindow(), but it seemed overly complicated and I did not like it's asyncronous approach. I prefer using pd.DataFrame(), so I stuck with that. The code below is simpler than above and now includes a DatetimeIndex rather than in an integer index. Only tested (so far) on Tiingo dailly data.
class DataAggregator: ''' A class for aggregating one field from a slice from external data source It collects a rolling window of n points tickers: List of tickers to extract field: String defining which field to extract, eg. 'AdjustedClose' n: number of points to keep ''' def __init__(self, tickers, field, n): self.tickers = tickers self.field = field self.n = n self.data = pd.DataFrame() def agg(self, slice): for t in self.tickers: self.data.loc[slice.Time, t] = getattr(slice[t], self.field) if len(self.data) > self.n: self.data.drop(self.data.index[0], inplace=True)
A Leo
Hi GregasMaximus  this is exactly what I was looking for.Â
I almost threw the towel on Quantconnect. Thank you for this post.Â
And I have a question: Â but I am curious, don't you need to authincate your access to TIINGO? how did you do this?
A Leo
Hi Jared Broad, I know this post is old and I have the question above (which I am hoping Gregas can answer if he still works with TIINGO data), I hope you can assist me with any pointer on how to work with TIINGO data as I could not find any documentation on it (other than this post). thank you.
Derek Melchin
Hi I Malak,
See the Tiingo News Feeds docs.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
A Leo
… and by the way even the example straight from the documentation (link below) does not work. all data requests failed. I am welling to spend a bit more time on this but I have to say it's a bit frustrating.Â
Derek Melchin
Hi A Leo,
We couldn't reproduce the issue. Note that a dataset subscription is required to request Tiingo News Feed data.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
GregasMaximus
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!