Share New Research 
So I am working with a dynamic universe of stocks at second resolution and am occasionally seeing the not found in tradebars object error. I am using the ~data.ContainsKey recommendation to skip the tradebar for stocks that show up as errors in the log. But I have two questions:
1. When dealing with a dynamic universe of stocks, I can't anticipate all the stocks that doesn't have data for a particular second. Right now I am listing all the stocks I am encountering with this error on a one week backtest. But if I do a 1 year backtest or am running live, how do I dynamically account for the issue?
if (data == null || !data.ContainsKey("CMCM") || !data.ContainsKey("SPY") || !data.ContainsKey("FL") || !data.ContainsKey("ROST")) return;
2. If there are many stocks in your universe and you avoid an OnData tick whenever any one of those stocks doesn't have data for a particular second, then won't that mean I will be missing many iterations of OnData? i.e. if it returns because 1 stock out of 100 in the universe doesn't have data for the second, then that means I am not running my OnData logic for the 99 stocks that are still available.
Yan Xiaowei
Hi Sofyan Saputra ,
The followings are my solutions to your issue:
1. You don't need to the symbol manually. My solution is to store the symbols you added to the universe winthin the selection functions, and then it's convenient for you to check whether they are missing in a data slice.
def CoarseSelection(self,coarse): Â Â sortedByDollarVolume = sorted(coarse, key=lambda x: Â Â Â Â x.DollarVolume, reverse=True) Â Â top = sortedByDollarVolume[20:] Â Â list = List[Symbol]() Â Â for x in top: Â Â Â Â list.Add(x.Symbol) # store your symbol here. Â Â self.symbols = [x.Symbol for x in top] Â Â return list def OnData(self, data): Â Â if False in [data.ContainsKey(x) for x in self.symbols]: Â Â Â Â return2. When you iterate through your list of symbols, you don't necessarily need to 'return'. You can use 'continue' to skip that iteration, or just remove that symbol. It depends on what do you want to do when encountering a missing data slice.
If you have any further questions, feel free to post here :)
Sofyan Saputra
Hi Yan Xiaowei, appreciate the assistance!
1. Great solution; I just implemented it in my code in C# as follows. Whereby _universeSymbols is a list of symbol strings I stored when creating my universe. I wonder if there is some way to get the list of symbols from the universe without an intermediary data structure.
if ((from symbol in _universeSymbols where !data.ContainsKey(symbol) select symbol).Count() > 0) return;2. I'm confused as to how I would implement this. What iteration of a loop am I using 'continue' on and what would the difference be with 'return' since OnData will be called again the next second anyways?
Also, how would one go about removing a symbol from your universe for one iteration of OnData, so you could process the other stocks in your universe?
Alexandre Catarino
The slice object has the member Keys where all the symbols in that object are listed.
So, instead of avoiding what is missing, we can work with what is included.
You can remove a security from the Universe with the RemoveSecurity(symbol) method. FYI: it also cancels open orders and liquidades positions.
Yan Xiaowei
Hi Sofyan Saputra
1. There is a way to get the list of symbols, but I'm sure you will find the intermediary data structure way more convenient.
self.symbols = [] key_symbols = self.UniverseManager.Keys for i in key_symbols: if str(i.Value) == "QC-UNIVERSE-COARSE-USA": self.uni_symbol = i for i in self.UniverseManager[self.uni_symbol].Members: self.symbols.append(i.Value.Symbol)2. Anywhere you use 'return' means the OnData function will end and returns nothing. When you iterate through your list of symbols, you can use continue, for example:
# self.symbols is your symbol list def OnData(self,data): for i in self.symbols: if not data.ContainsKey(i): self.Log('%s is missing'%str(i)) continue # Alternatively, you can choose to remove it: self.symbols.remove(i) #do what you want here.Sofyan Saputra
That makes perfect sense, thanks for the help! I am only using onData to simulate a trailing stop for my orders, so instead of looping through all the securities, I am only looping through the ones I have stop orders for and continuing whenever I am missing data for any particular stock in my list of stop orders. The code looks like below.
It seems like the results of the backtest are significantly different (worse) when I implement OnData this way. It appears more accurate since I am not throwing out many seconds of data using a return and thus my trailing stops are actually filling. This is why I wish the trailing stop loss was an actual order type we could backtest because in real life, the stop order may fill while the backtest may miss some data points where the stop order should fill. I tried using tick data but it results in extremely slow backtests.
foreach (OrderTicket stopTicket in stopTickets ?? Enumerable.Empty<OrderTicket>()) { if ((from s in _universeSymbols where !data.ContainsKey(s) select s).ToList().Contains(stopTicket.Symbol)) continue; //...rest of my logic }Sofyan Saputra
Actually, I just finished the test using tick data and am geting "Runtime Error: '...' wasn't found in the TradeBars object" when I do a continue in my list of stocks with stopOrders. But I am getting no errors when I just return at the top of ondadata. Is it trying to get the data at some point before I do the continue? I am not really seeing where that is happening?
public void OnData(TradeBars data) { if (IsWarmingUp) return; if (!canPoll) return; //If I do a return here, the code works without errors //if ((from symbol in _universeSymbols where !data.ContainsKey(symbol) select symbol).Count() > 0) return; // Simulate trailing stop loss foreach (OrderTicket stopTicket in stopTickets ?? Enumerable.Empty<OrderTicket>()) { //If I individually continue any missing stocks, it doesn't work if ((from s in _universeSymbols where !data.ContainsKey(s) select s).ToList().Contains(stopTicket.Symbol)) continue; decimal price = data[stopTicket.Symbol].Price; int dir = -1 * stopTicket.Quantity > 0 ? 1 : -1; decimal currStopPrice = stopTicket.Get(OrderField.StopPrice); decimal newStopprice = calcStop(dir, price); // Improve the stop if the price gets better if ((dir == 1 && newStopprice > currStopPrice) || (dir == -1 && newStopprice < currStopPrice)) { stopTicket.Update(new UpdateOrderFields { StopPrice = newStopprice }); } } }Alexandre Catarino
We can get all the stop ticket from the Transactions object:
stopTickets = Transactions .GetOrderTickets(x => x.OrderType == Orders.OrderType.StopMarket);Jared Broad
Please if possible Sofyan Saputra attach backtests not code snippets. It makes debugging assistance much more productive.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Sofyan Saputra
Jared Broad Apologies for this. The main issue deterring me from uploading backtests is that this would put my algorithm in public domain, which I am not looking to do. However, in the future I will try to input the snippets into a mini-seperate-backtest devoid of the other algorithm's logic to make debugging more productive.
Sofyan Saputra
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!