Share New Research 
Hi guys,
I'm getting an error when backtesting that is unexpected because I have tried to make the algo resillient to this exact scenario.
in the initialization phase set the following:
self.tickers = ['ESS','STOR'] in the OnData phase of the algo I start with
if False in [data.ContainsKey(x) for x in self.tickers]: returnand i have also tried
if not all([data.ContainsKey(x) for x in self.tickers]): returnin both instances I get an KeyError (full error message at the bottom on 26 september 2019 for ticker 'ESS' (I have had the same error for other tickers on other dates). This specific example is with minute resolution data.
I would expect that if the ticker data is not present the algo would just skip this data point and continue with the next datapoint.
anyone running into something similar or am I missing an elephant in the room?
Best,
Dirk
Runtime Error: KeyNotFoundException : 'ESS' wasn't found in the TradeBars object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("ESS")
at QuantConnect.Data.Market.DataDictionary`1[T].get_Item (QuantConnect.Symbol symbol) [0x00024] in <8e85730333624c7d90e25a24998a3d52>:0
at QuantConnect.Data.Market.TradeBars.get_Item (QuantConnect.Symbol symbol) [0x00000] in <8e85730333624c7d90e25a24998a3d52>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0
at OnData in main.py:line 35
:: self.pair.updateMetrics(data)
at updateMetrics in main.py:line 96
KeyNotFoundException : 'ESS' wasn't found in the TradeBars object, likely because there was no-data at this moment in time and it wasn't possible to fillforward historical data. Please check the data exists before accessing it with data.ContainsKey("ESS")
at QuantConnect.Data.Market.DataDictionary`1[T].get_Item (QuantConnect.Symbol symbol) [0x00024] in <8e85730333624c7d90e25a24998a3d52>:0
at QuantConnect.Data.Market.TradeBars.get_Item (QuantConnect.Symbol symbol) [0x00000] in <8e85730333624c7d90e25a24998a3d52>:0
at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00032] in <b0e1ad7573a24fd5a9f2af9595e677e7>:0 (Open Stacktrace)
Alexandre Catarino
Hi dirk bothof ,
Could you please share a backest that we can use to reproduce the issue?
Dirk bothof
I do not believe it is possible to attach a failed backtest, so I have attached the algorithm below, copy paste should work :) (yes, this will lose money pretty fast ;)
import pandas as pd import numpy as np class Trader(QCAlgorithm): def Initialize(self): self.SetBrokerageModel(BrokerageName.InteractiveBrokersBrokerage) self.SetStartDate(2019,8,6) # Set Start Date #self.SetEndDate(2019,9,10) self.SetCash(50000) # Set Strategy Cash self.tickers = ['ESS','STOR'] self.enter_threshold = 2 self.exit_threshold = -self.enter_threshold self.enter_size = 0.5 self.pair = Pair(self.AddSecurity(SecurityType.Equity, self.tickers[0], Resolution.Minute).Symbol, self.AddSecurity(SecurityType.Equity, self.tickers[1], Resolution.Minute).Symbol ) #setting custom slippage model for ticker in self.tickers: self.Securities[ticker].SetSlippageModel(CustomSlippageModel(self)) self.SetWarmUp(self.pair.lookback) def OnData(self, data): '''OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here. ------- data: Slice object keyed by symbol containing the stock data ''' if False in [data.ContainsKey(x) for x in self.tickers]: return #updating the pairs self.pair.updateMetrics(data) if self.IsWarmingUp: return #no trading during warm up period #potentially buying stock if we don't hold any if self.pair.position == 0: if self.pair.zScore < -self.enter_threshold: #check if we need to go long or short on the spread self.SetHoldings(self.pair.a, self.enter_size) self.SetHoldings(self.pair.b, -self.enter_size) self.pair.position += -1 self.Log(self.pair.distanceRatio) elif self.pair.zScore > self.enter_threshold: self.SetHoldings(self.pair.a, -self.enter_size) self.SetHoldings(self.pair.b, self.enter_size) self.pair.position += 1 self.Log(self.pair.distanceRatio) else: pass #potentially selling stocks if we already have position elif (self.pair.position == -1 and self.pair.zScore > self.exit_threshold) or (self.pair.position == 1 and self.pair.zScore < -self.exit_threshold): self.Liquidate(self.pair.a) self.Liquidate(self.pair.b) self.pair.position = 0 self.Log(self.pair.distanceRatio) #no trading oppertunities found pass else: pass class Pair(object): def __init__(self,ticker_a,ticker_b): '''initializes the algoritme, assigns variables, and loads a bit of historical data so the algo is warmed up --- ticker_a: ticker of the first stock of the pair ticker_b: ticler of the second stock of the pair ''' self.a = ticker_a self.b = ticker_b self.position = 0 self.lookback = 8*330 #initiating trading signal self.price_a = np.array(0) self.price_b = np.array(0) self.ratio = np.array(0) self.sma = 0 self.meanDistance = 0 self.distanceRatio = 0 self.zScore = 0 def updateMetrics(self,data): '''Updates the metrics of the inserted pair with the data of the Slice Object Keeps memory use small, and uses lists for all opperations (df or series would not work since there is not vector caluclations) data: data object in the quantconnect model ''' #making/updating metrics self.price_a = np.append(self.price_a, (data.Bars[self.a]).Close) self.price_b = np.append(self.price_b, (data.Bars[self.b]).Close) self.ratio = np.append(self.ratio, self.price_a[-1]/self.price_b[-1]) if self.price_a.shape[0] > self.lookback: #if the data list is longer than lookback the list is trimmed to not impact 'rolling' stats self.price_a = self.price_a[1:] self.price_b = self.price_b[1:] self.ratio = self.ratio[1:] if self.ratio.shape[0] >= self.lookback: self.std = np.std(self.ratio) self.sma = np.mean(self.ratio) self.zScore = (self.ratio[-1] - self.sma) / self.std class CustomSlippageModel: def __init__(self, algorithm): self.algorithm = algorithm def GetSlippageApproximation(self, asset, order): # custom slippage math slippage = asset.Price * 0.0001 * np.log10(2*float(order.AbsoluteQuantity)) return slippageAlexandre Catarino
Hi dirk bothof ,
Please use
if not all([data.Bars.ContainsKey(symbol) for symbol in self.tickers]): returnsince the algorithm is trying to update the state of the Pair object using data.Bars:
self.price_a = np.append(self.price_a, (data.Bars[self.a]).Close) self.price_b = np.append(self.price_b, (data.Bars[self.b]).Close)A given Symbol can be in data.Keys but not in data.Bars.Keys if there is no TradeBar data.
In this case, there was only dividend data when it tried to access TradeBar data.
Alvin lim
Hi Alexandre,
Thanks for the code snippet to address this issue as I encountered it as well.
However it uncovers another issue where all the symbols are not found in the bars.
Can I check what will result in such a scenario? I suspect I am doing something incorrect.
Rahul Chowdhury
Hi Alvin,
A symbol can be found in data.Keys but not in data.Bars.Keys if there is no TradeBar data for that time step.
This can happen if there is no TradeBar data for that symbol while there is dividend data, splits data, etc.
If you need help with something specific, please post a code snippet which reproduces the problem and we'll be glad to check it out.
Best
Rahul
Dirk bothof
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!