Hello QuantConnect Community! I need help with this code. What I'm trying to do is:
a. Consolidate minute bars over longer periods.
b. Then create a SMA of consolidated bars and make a rolling window with those bars.
c. Make another rolling window of the SMA consolidated bars in order to be able to compare the price of the consolidated bars with the SMA and execute trades.
So I took as reference the Github code: https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/MultipleSymbolConsolidationAlgorithm.py and the change I made was to remove the manual universe and I was using a Coarse Selection Filter to avoid human bias. But I get this error in OnDataConsolidated function:
Runtime Error: Trying to retrieve an element from a collection using a key that does not exist in that collection throws a KeyError exception. To prevent the exception, ensure that the AAPL key exist in the collection and/or that collection is not empty. at OnDataConsolidated self.Data[bar.Symbol.Value].SMA.Update(bar.Time in main.py: line 76
I searched on the forum discussions and didn't find this KeyError addressed with a universe selections filter.
Also, I’m not sure if the Rolling window of the bars is working well in the following lines of code:
a = format(self.Data[symbol].Bars[0])
b = format(self.Data[symbol].SMA[1])
if a >= b:
self.MarketOrder(symbol, 100)
continue
And last I don’t quite understand what “format” makes prior to:
format(self.Data[symbol].Bars[0])
Any help will be appreciated. I attached the code.
from System import *
from QuantConnect import *
from QuantConnect.Data.Consolidators import *
from QuantConnect.Data.Market import *
from QuantConnect.Orders import OrderStatus
from QuantConnect.Algorithm import QCAlgorithm
from QuantConnect.Indicators import *
import numpy as np
from datetime import timedelta, datetime
class MultipleSymbolConsolidationAlgorithm(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2021, 1, 26) # Set Start Date\
self.SetEndDate(2021, 4, 26)
# Holds all of our data keyed by each symbol
self.Data = {}
# Contains all of our equity symbols
self.AddUniverse(self.CoarseSelectionFilter)
self.UniverseSettings.Resolution = Resolution.Minute
self.UniverseSettings.SetDataNormalizationMode = DataNormalizationMode.Raw
self.UniverseSettings.Leverage = 1
self.__numberOfSymbols = 10
def CoarseSelectionFilter(self, coarse):
sortedByDollarVolume = sorted(coarse, key=lambda c: c.Volume, reverse=True)
filteredByPrice = [c.Symbol for c in sortedByDollarVolume if c.Price > 10 ]
return filteredByPrice[:self.__numberOfSymbols]
def OnSecuritiesChanged(self, changes):
# This is the period of bars we'll be creating
BarPeriod = TimeSpan.FromMinutes(10)
# This is the period of our sma indicators
SimpleMovingAveragePeriod = 20
# This is the number of consolidated bars we'll hold in symbol data for reference
RollingWindowSize = 10
for security in changes.AddedSecurities:
symbol = security.Symbol
if symbol not in self.Data:
self.Data[security] = SymbolData(symbol, BarPeriod, RollingWindowSize)
# loop through all our symbols and request data subscriptions and initialize indicator
for symbol, symbolData in self.Data.items():
# define the indicator
symbolData.SMA = SimpleMovingAverage(self.CreateIndicatorName(symbol.Symbol, "SMA" + str(SimpleMovingAveragePeriod), Resolution.Minute), SimpleMovingAveragePeriod)
# define a consolidator to consolidate data for this symbol on the requested period
consolidator = TradeBarConsolidator(BarPeriod)
# write up our consolidator to update the indicator
consolidator.DataConsolidated += self.OnDataConsolidated
# we need to add this consolidator so it gets auto updates
self.SubscriptionManager.AddConsolidator(symbolData.Symbol, consolidator)
for security in changes.RemovedSecurities:
symbol = security.Symbol
if symbol in self.Data:
symbolData = self.Data.pop(symbol, None)
self.SubscriptionManager.RemoveConsolidator(security.Symbol, consolidator)
if security.Invested:
self.Liquidate(security.Symbol, "Universe Removed Security")
def OnDataConsolidated(self, sender, bar):
self.Data[bar.Symbol.Value].SMA.Update(bar.Time, bar.Close)
if self.Data[bar.Symbol.Value].SMA.IsReady:
sma = self.Data[bar.Symbol.Value].SMA.Current.Value
self.Data[bar.Symbol.Value].HistoricalSMA.Add(sma)
self.Data[bar.Symbol.Value].Bars.Add(bar)
# OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
# Argument "data": Slice object, dictionary object with your stock data
def OnData(self,data):
# loop through each symbol in our structure
for symbol in self.Data.keys():
symbolData = self.Data[symbol]
# this check proves that this symbol was JUST updated prior to this OnData function being called
if symbolData.IsReady() and symbolData.WasJustUpdated(self.Time):
self.Debug("Data rolling window".format(self.Data[symbol].SMA))
if not self.Portfolio[symbol].Invested:
self.SetHoldings(symbol, 0.1)
self.Debug("Data: {0} -> {1}".format(self.Data[symbol].Bars[1],self.Data[symbol].SMA))
'''------------------------- Trading logic ----------------------------------- '''
a = format(self.Data[symbol].Bars[0])
b = format(self.Data[symbol].SMA[1])
if a >= b:
self.MarketOrder(symbol, 100)
continue
if a <= b:
self.MarketOrder(symbol, -100)
continue
class SymbolData(object):
def __init__(self, symbol, barPeriod, windowSize):
self.Symbol = symbol
# The period used when population the Bars rolling window
self.BarPeriod = barPeriod
self.Bars = RollingWindow[IBaseDataBar](windowSize)
# The simple moving average indicator for our symbol
self.SMA = None
sma_lookback = 10
self.HistoricalSMA = RollingWindow[float](sma_lookback)
# Returns true if all the data in this instance is ready (indicators, rolling windows, ect...)
def IsReady(self):
return self.Bars.IsReady and self.SMA.IsReady
# Returns true if the most recent trade bar time matches the current time minus the bar's period, this
# indicates that update was just called on this instance
def WasJustUpdated(self, current):
return self.Bars.Count > 0 and self.Bars[0].Time == current - self.BarPeriod
Louis Szeto
Hi Yaros
The reason causing the error is that in line 51, the dictionary key is a Security object, while in line 76 it was called by a string object of its symbol. Therefore, the key does not exist.
For the second query, the rolling window value does not need format() to call. In fact, .format is just a method for replacing placeholders with values. Since it is storing a Bar, it is necessary to include a further attribute (e.g. .Close) for mathematical operation later. Also, the value of SMA has to be called by the HistoricalSMA rolling window:
We would also suggest putting line 55 to 63 into the SymbolData class to avoid redundant initializations. Please find the attached backtest for reference of the above.
Best
Louis Szeto
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Yaros
Hi Louis thank you very much for all your help, we realy appreciate it.
I have a couple of inquiries if you don't mind
a. I wonder if indicators have a warm-up period since no history data is called or is it something that happens automatically when setting up a rolling window?
b. we are trying to set up an schedure event for the universe selection to happen just once right at the moment of the backtest start date, we tried many ways to achive this but don't see how could achive this, we reviewed in the forum discussions and also checked out the Github schedure universe selection algorithm https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/ScheduledUniverseSelectionModelRegressionAlgorithm.py but nothing seems to fit owr needs.
This is what we tried so far in Initialize section:
1st attempt
and got this error message:
During the algorithm initialization, the following exception has occurred: TypeError : 'MultipleSymbolConsolidationAlgorithm' object is not iterableat CoarseSelectionFiltersortedByDollarVolume = sorted(coarse in main.py: line 44 TypeError : 'MultipleSymbolConsolidationAlgorithm' object is not iterable
2nd attempt
and we got pretty much the same error message
Runtime Error: TypeError : 'datetime.datetime' object is not iterableat CoarseSelectionFiltersortedByDollarVolume = sorted(coarse in main.py: line 44 (Open Stacktrace)
3rd attemp
we tried setting up some "if" conditionals in the CoarseSelectionFilter funtion but nothing seems to work well for us,
c. We would like the universe selections runs only once per week in an specific day of the week and other version to run once per month in an specifit week of the moth but we continue getting the same error message above when customizing the date and time rules.
d. and last how could find out all methods available for the LEAN properties for instance self.Time
Thanks a lot in advance for all your help
Varad Kabade
Hi Yaros,
a. All indicators need to be warmed up before using we set the Warmup period when we initialize the indicator. To check whether an indicator is ready use the IsReady property of the indicator.b,c. Would you please attach the snippet where the error is occurring inside the coarse function? The time rules and date rules inside the ScheduledUniverse selection are similar to scheduled events. Refer to the following doc for more information.
d. self.Time is a Python DateTime object.
Best,
Varad Kabade
Yaros
Hi Vadad thank you for replying, here attached the back test.
We tried using he code in the lines 25 and 26
and also with the code in the line 30 which is commented out in the last back test but give us the same error message
the snippet of code in which the error happens is in line 41
And about the warm-up inquiry, we do verify if the indicators are ready however we would like to have them ready from the beggining of the trading day and not having to wait until the indicator is ready since in this case bars are consolidated in 10 min bars and 20 SMA period which in time is 200 min beforre the SMA indicator gets ready.
Would you help us guiding us in wich line should we call the History to warm them up and how to implement it please.
Varad Kabade
Hi Yaros,
The selector method in the ScheduledUniverseSelection method does not receive the coarse object as input but receives a DateTime object.Would you help us guiding us in wich line should we call the History to warm them up and how to implement it please.
We recommend warming up the indicators inside the SymbolData class i.e add the following code snippet inside the
Best,
Varad Kabade
Yaros
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!