Error Consolidating 1 day & 30 minute consolidators

The error message you're seeing is due to the fact that you're trying to consolidate data into a smaller time frame (30 minutes) than the data you're receiving (1 day). In other words, you're trying to create a 30-minute bar from daily data, which is not possible.

In your code, you're creating two consolidators: one for 30 minutes and one for 1 day. However, the data you're receiving is likely daily data, which is why you're seeing this error.

If you want to create a 30-minute consolidator, you need to ensure that you're subscribing to data with a resolution of 30 minutes or less. For example, if you're subscribing to minute data, you can then consolidate this into 30-minute bars.

Here's an example of how you might subscribe to minute data and create a 30-minute consolidator:

class SymbolData:
    def __init__(self, algorithm, symbol):
        self.algorithm = algorithm
        self.symbol = symbol

        self.boundary = RollingWindow[TradeBar](20)
        self.consolidator_two = TradeBarConsolidator(timedelta(minutes=30))
        self.consolidator_two.DataConsolidated += self.OnDataConsolidated_two
        algorithm.SubscriptionManager.AddConsolidator(symbol, self.consolidator_two)

        self.Bars = RollingWindow[TradeBar](12) #defines rolling window size, TradeBar defines bar type
        self.consolidator = TradeBarConsolidator(timedelta(days=1))
        self.consolidator.DataConsolidated += self.OnDataConsolidated
        algorithm.SubscriptionManager.AddConsolidator(symbol, self.consolidator)

    def OnDataConsolidated_two(self, sender, bar):
        self.boundary.Add(bar)

    def OnDataConsolidated(self, sender, bar):
        self.Bars.Add(bar)

    @property
    def IsReady_two(self):
        return self.boundary.IsReady

    @property
    def IsReady(self):
        return self.Bars.IsReady

In your main algorithm, when you add the security, make sure to specify the resolution as minute:

self.AddEquity("ABCL", Resolution.Minute)

This will ensure that you're receiving minute data, which can then be consolidated into 30-minute bars.

Hi Mia Alissi-

I'm using 1 minute data as inputs to the algo. Here is the initialize function. I'm still confused as to why it thinks I'm passing daily bars into the consolidator when my universe setting is minute.

class TradeStrategyTest(QCAlgorithm):
  
  def Initialize(self):
       self.SetStartDate(2023,3, 1)  
       self.SetEndDate(2023,6,22) 
       self.SetCash(100000000)
       self.positionsize = 100       
       self.ticket = 0  

       self.Data = {}
       self.AddEquity("SPY", Resolution.Daily).Symbol
       self.rebalanceTime = datetime.min

       self.AddUniverse(self.CoarseSelectionFilter, self.FineSelectionFilter)
       self.UniverseSettings.Resolution = Resolution.Minute                                                  
       self.SetWarmUp(200, Resolution.Daily) # pumps in data ahead of the start time of the algorithm, with this method indicators will be ready on the first day at start
       # warmup will appear on plots on a daily resolution, even if incoming universe is finer
       

The error message you're seeing is due to the fact that you're trying to consolidate data into a smaller time frame than the data's resolution. In your Initialize method, you're adding "SPY" with a daily resolution:

self.AddEquity("SPY", Resolution.Daily).Symbol

Then, in your SymbolData class, you're trying to create a 30-minute consolidator:

self.consolidator_two = TradeBarConsolidator(timedelta(minutes=30))

This is not possible because you can't create 30-minute bars from daily data. You need to ensure that the resolution of your data is the same as or smaller than the period of your consolidator.

If you want to use a 30-minute consolidator, you should add "SPY" with a minute or second resolution. For example:

self.AddEquity("SPY", Resolution.Minute).Symbol

Also, make sure to remove the consolidator when the security leaves the universe. If you have a dynamic universe and don't remove consolidators, they compound internally, causing your algorithm to slow down and eventually die once it runs out of RAM. You can remove the consolidator using the RemoveConsolidator method:

self.SubscriptionManager.RemoveConsolidator(self.symbol, self.consolidator)

For more information, you can refer to the QuantConnect documentation on consolidators.

Hi Mia-

I did both of those things with no improvement. Here is the complete code to backtest and debug, if you would like. 

Could it be due to the warmup period being in days?

self.SetWarmUp(200, Resolution.Daily)

#region imports
from AlgorithmImports import *
#endregion
from datetime import datetime, date, time
from datetime import timedelta
import math
import numpy as np
from sklearn.linear_model import LinearRegression
import statistics

#   indicators and history requests are the slowest part of the algorithm

class TradeStrategyTest(QCAlgorithm):
  
  def Initialize(self):
       self.SetStartDate(2023,3, 1)  
       self.SetEndDate(2023,6,22) 
       self.SetCash(100000000)
       self.positionsize = 100       
       self.ticket = 0  
       self.mystring = 0
       self.mystring_two = 0
       self.mystring_total = 0
       self.myvar = 0
       self.myshort = 0
       #self.dict_ready = 0

       self.banned = {'AAC', 'BCE', 'CVE', 'AMTD', 'SPY', 'CNI', 'CPG', 'PTGI', 'VATE', 'ANIX', 'KMPH', 'FLNT', 'PME', 'SPCB'} #add problematic symbols to this dict which be added to Data
       self.dummy_dict = {}
       self.sma_dict = {} #stores SMA values from the manual indicator
       self.RSI = {}
       self.long_sma = {}
       self.fifty_wk = {}
       self.fifty_min = {}
       self.beta = {}
       self.smacounter = {}
       self.nasdaqratio = {}
       self.nasdaqave = 0
       self.earningdate = {}
       self.Targets = {} 
       self.open_position = {}
       self.AskVol = {}
       self.high_zscore = {}

       self.add_one = 0
       self.add_two = 0
       self.nasdaqCode = "FINRA/FNSQ_"  #NASDAQ
       self.nyseCode = "FINRA/FNYX_"  #NYSE

       self.windowlength = 12
       self.mylow = 0

       self.Data = {}
       self.AddEquity("SPY", Resolution.Daily).Symbol
       self.rebalanceTime = datetime.min
       self.SetBrokerageModel(BrokerageName.InteractiveBrokersBrokerage, AccountType.Cash) # $1.29/order average for IB. IB trading equities defaults to T+2 settlement
       self.Settings.FreePortfolioValuePercentage = 0.025
       self.AddUniverse(self.CoarseSelectionFilter, self.FineSelectionFilter)
       self.UniverseSettings.Resolution = Resolution.Minute                                                  
       self.SetWarmUp(200, Resolution.Daily) # pumps in data ahead of the start time of the algorithm, with this method indicators will be ready on the first day at start
       # warmup will appear on plots on a daily resolution, even if incoming universe is finer
       
       self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.BeforeMarketClose("SPY", 1), self.checkpositions) #contains price close to true close *but different*,, contains SMA from previous day
       #self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.Midnight, self.ClearFills) #contains true Close of current day, contains SMA from previous day
       #self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.At(9, 30), self.firstcheck) # doesn't come in until 9:31, has identical SMA and price as midnight, contains true Close of previous day, redundant to midnight
       self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.At(9, 31), self.scantargets) #first unique price of the new trading day from the close of the first minute bar (comes at same time as 9:30), arrives at 9:31
       #self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.At(9, 32), self.thirdcheck) #9:32 is when the previous day's SMA arrives
       

  def OnData(self, data):

        for symbol in self.Data.keys():
            symbolData = self.Data[symbol]

            if symbol.Value in self.Targets.keys():
                if not symbolData.IsReady: 
                    continue    
                pass


  def scantargets(self):

        self.Targets = {}
        for symbol in self.Data.keys():
            symbolData = self.Data[symbol]

            if not symbolData.IsReady: 
                continue
            
  def checkpositions(self):

    for symbol in self.Data.keys():
        symbolData = self.Data[symbol]

        if symbol.Value in self.Targets.keys():
            pass

  def CoarseSelectionFilter(self, coarse): #updates at midnight during backtesting, and around 6-7am in live trading, based on previous day data

        symbols_by_price = [c for c in coarse if c.Price > 5 and c.Price < 10 and c.HasFundamentalData and c.Volume > 100000 and len(c.Symbol.Value) <= 4 and c.Market == 'usa']

        self.filteredByPrice = symbols_by_price[:20]

        return [c.Symbol for c in self.filteredByPrice]

  def FineSelectionFilter(self, fine):
        sortedByMarketCap = sorted(fine, key=lambda c: c.MarketCap)
        symbols_by_marketcap = [c for c in sortedByMarketCap if c.MarketCap > 0 and c.CompanyReference.IsREIT is False and (self.Time - c.SecurityReference.IPODate)  > timedelta(days=365)]
        self.filteredBymarketcap = symbols_by_marketcap

        return [c.Symbol for c in self.filteredBymarketcap]

  def OnSecuritiesChanged(self, changes):
        for security in changes.AddedSecurities:
            symbol = security.Symbol
            if symbol not in self.Data and symbol.Value not in self.banned:
                self.Data[symbol] = SymbolData(self, symbol)
                

                if symbol in self.Data.keys():
                
                    self.sma_dict[symbol] = SimpleMovingAverage(13) #define the type of indicator and # of periods
                    self.history = self.History(symbol, 13, Resolution.Daily).loc[symbol] #warm up doesn't work in a universe, need to use history to grab data
                    if not self.history.empty:    
                        for idx, row in self.history.iterrows(): #grab close price data
                            self.sma_dict[symbol].Update(idx, row['close'])
                    
                    self.RegisterIndicator(symbol, self.sma_dict[symbol], Resolution.Daily) #register the sma indicator, define daily data

                    self.long_sma[symbol] = SimpleMovingAverage(200) #define the type of indicator and # of periods
                    self.history = self.History(symbol, 200, Resolution.Daily).loc[symbol] #warm up doesn't work in a universe, need to use history to grab data
                    if not self.history.empty:  
                        for idx, row in self.history.iterrows(): #grab close price data
                            self.long_sma[symbol].Update(idx, row['close'])
                    self.RegisterIndicator(symbol, self.long_sma[symbol], Resolution.Daily) #register the sma indicator, define daily data

                    self.RSI[symbol] = RelativeStrengthIndex(14) #define the type of indicator and # of periods
                    self.history = self.History(symbol, 14, Resolution.Daily).loc[symbol] #warm up doesn't work in a universe, need to use history to grab data
                    if not self.history.empty:  
                        for idx, row in self.history.iterrows(): #grab close price data
                            self.RSI[symbol].Update(idx, row['close'])
                    self.RegisterIndicator(symbol, self.RSI[symbol], Resolution.Daily) #register the sma indicator, define daily data

                    self.fifty_wk[symbol] = Maximum(250) #in trading days = 1 calender year, grabs the close price as the high
                    
                    self.history = self.History(symbol, 250, Resolution.Daily).loc[symbol] #warm up doesn't work in a universe, need to use history to grab data
                    if not self.history.empty: 
                        for idx, row in self.history.iterrows(): #grab close price data
                            self.fifty_wk[symbol].Update(idx, row['close'])
                    self.RegisterIndicator(symbol, self.fifty_wk[symbol], Resolution.Daily) #register the sma indicator, define daily data
                
                    self.fifty_min[symbol] = Minimum(250) #in trading days = 1 calender year, grabs the close price as the low
                    self.history = self.History(symbol, 250, Resolution.Daily).loc[symbol] #warm up doesn't work in a universe, need to use history to grab data
                    if not self.history.empty:  
                        for idx, row in self.history.iterrows(): #grab close price data
                            self.fifty_min[symbol].Update(idx, row['close'])
                    self.RegisterIndicator(symbol, self.fifty_min[symbol], Resolution.Daily) #register the sma indicator, define daily data

                    self.smacounter[symbol.Value] = [0]
                    self.nasdaqratio[symbol.Value] = [0]
                    self.earningdate[symbol.Value] = [0]
                    self.Targets[symbol.Value] = [0]
                    self.open_position[symbol.Value] = [0]
                    self.AskVol[symbol.Value] = [0]
                    self.high_zscore[symbol.Value] = [0]

        for security in changes.RemovedSecurities:
            symbol = security.Symbol
            if symbol in self.Data:
                symbolData = self.Data.pop(symbol, None)
                self.SubscriptionManager.RemoveConsolidator(symbol, symbolData.consolidator)
                self.SubscriptionManager.RemoveConsolidator(symbol, symbolData.consolidator_two)
                self.sma_dict.pop(symbol) #removes symbols from the sma dict 
                self.RSI.pop(symbol)
                self.long_sma.pop(symbol)
                self.fifty_wk.pop(symbol)
                self.fifty_min.pop(symbol)
                # *** self.beta.pop(symbol)

class NasdaqCustomColumns(NasdaqDataLink):#
    def __init__(self):
        self.ValueColumnName = "ShortVolume"

class SymbolData:
    def __init__(self, algorithm, symbol):
        self.algorithm = algorithm
        self.symbol = symbol

        self.boundary = RollingWindow[TradeBar](20)
        self.consolidator_two = TradeBarConsolidator(timedelta(minutes=30))
        self.consolidator_two.DataConsolidated += self.OnDataConsolidated_two
        algorithm.SubscriptionManager.AddConsolidator(symbol, self.consolidator_two)

        self.Bars = RollingWindow[TradeBar](12) #defines rolling window size, TradeBar defines bar type
        self.consolidator = TradeBarConsolidator(timedelta(days=1))
        self.consolidator.DataConsolidated += self.OnDataConsolidated
        algorithm.SubscriptionManager.AddConsolidator(symbol, self.consolidator)
        
    def OnDataConsolidated_two(self, sender, bar):
        self.boundary.Add(bar)

    def OnDataConsolidated(self, sender, bar):
        self.Bars.Add(bar)

    @property
    def IsReady_two(self):
        return self.boundary.IsReady

    @property
    def IsReady(self):
        return self.Bars.IsReady

Solved and will close discussion. 

Solution: I needed to change the warmup resolution from Daily to Minute as it was trying to consolidate the daily warmup data into minutes. Changed to:

self.SetWarmUp(200, Resolution.Minute)

Based on the information provided, it seems like 'Justin' was trying to consolidate daily data into 30-minute bars, which was causing an error. The error message suggests that the data cannot be consolidated into a smaller time frame than the original data.

In this case, 'Justin' solved the issue by changing the warmup resolution from Daily to Minute. This allowed the algorithm to have minute-level data, which can be consolidated into 30-minute bars. Here is the code 'Justin' used to change the warmup resolution:

self.SetWarmUp(200, Resolution.Minute)

This code sets the warmup period to 200 minutes, which provides enough minute-level data for the 30-minute consolidator.

Remember, when using consolidators, the resolution of the data you're consolidating should be the same or smaller than the resolution of the consolidator. For example, you can consolidate minute data into 30-minute bars, but you can't consolidate daily data into 30-minute bars.

Also, when adding an event handler to the consolidator, remember to pass the name of your method to the event system without parenthesis. If you use parenthesis, the method executes and the result is passed as the event handler instead of the method itself. For example, it should be self.consolidation_handler, not self.consolidation_handler().

Here is an example of how to add an event handler to a consolidator:

self.consolidator.DataConsolidated += self.consolidation_handler

In this line, self.consolidation_handler is the method that will handle the consolidated data.