Hi Sheikh,

Thank you for sharing this awesome algorithm. To further improve, we suggest you use minute resolution for more accurate filling prices, with daily checking conditions updated by daily consolidator.

Also, we would suggest using trade bar data: data.Bars.ContainsKey("SPY") and data.Bars["SPY"].Close in the original backtest to avoid error using only data["SPY"] if no bar data received. Please check docs for details.

Best,
Louis Szeto

Thanks Louis! I only work on Daily strategies so I will go through what I have and update them with this method of using a daily consolidator on minute data. Also, I'm glad you suggested it as this might be the way for me to develop daily Futures strategies using Futures minute data. Cool.

Hi Louis, as mentioned I am going back and changing my daily algos to use the method you suggested of consolidating the minute data into daily bars. This algo, code below, is running very slowly and outputting a minute equity curve, rather than daily. There are a couple of indicators also, so I was wondering how to get the indicators to use the daily bars consolidated from the minute data rather than straight from the Resolution.Daily data? The strategy is also using multiple crypto symbols so can you kindly check if I am implementing multiple symbols correctly? Thanks very much for your help!

class StatArbAlgo(QCAlgorithm):
    
    def Initialize(self):
        
        self.SetStartDate(2016, 5, 2)  
        self.SetEndDate(2021, 7, 14)  
        self.SetCash(1000000)  
        self.AddEquity("SPY", Resolution.Daily)  
        self.SetBenchmark("SPY")
        self.SetBrokerageModel(BrokerageName.AlphaStreams)
        self.SetExecution(ImmediateExecutionModel())
        self.SetPortfolioConstruction(EqualWeightingPortfolioConstructionModel())
        
        symbols = ["BTCUSD","ETHUSD","LTCUSD"]
        for symbol in symbols:
            self.AddCrypto(symbol)
            
        self.AddUniverseSelection(ManualUniverseSelectionModel(symbols))
        
        self.AddAlpha(StatArbAlphaModel(self, symbols))
                        
class StatArbAlphaModel(AlphaModel):
                        
    def __init__(self, algorithm, symbols):
        self.algo = algorithm
        self.symbols = symbols
        self.dataBySymbol = {}
        for symbol in self.symbols:
            self.dataBySymbol[symbol] = SymbolData(self.algo, symbol)
        self.period = 30
        
    def Update(self, algorithm, data):
        insights = []
        
        for symbol in self.symbols:
            if self.dataBySymbol[symbol].Updated:
                if data.Bars.ContainsKey(symbol) and self.dataBySymbol[symbol].IsReady():
                    
                    if self.dataBySymbol[symbol].slow < self.dataBySymbol[symbol].fast:
                        insights.append(Insight(symbol, timedelta(days=30), InsightType.Price, InsightDirection.Up))
                    else:
                        insights.append(Insight(symbol, timedelta(days=30), InsightType.Price, InsightDirection.Flat))
                    
        return insights
    
    def OnSecuritiesChanged(self, algorithm, changes):
        self.changes = changes 
        
class SymbolData:
    def __init__(self, algorithm, symbol):
        self.algo = algorithm
        self.symbol = symbol
        
        self.algo.Consolidate(symbol, Resolution.Daily, self.DailyBarHandler)
        
        self.open = 0
        self.close = 0
        self.high = 0
        self.low = 0
        self.Updated = False
        
        self.fast = self.algo.SMA(self.symbol, 5, Resolution.Daily, Field.Low)
        self.slow = self.algo.SMA(self.symbol, 60, Resolution.Daily, Field.Low)
        
    def DailyBarHandler(self, consolidated):
        self.open = consolidated.Open
        self.close = consolidated.Close
        self.high = consolidated.High
        self.low = consolidated.Low
        self.Updated = True
            
    def IsReady(self):
        return self.fast.IsReady and self.slow.IsReady
        
+ Expand
- Collapse

Hi Sheikh

It is very slow as you're running Update on alpha every minute. You may use a timestamp variable to avoid very close updating. We've attached a backtest for your reference.

Few more things to mention:

• You have to warm up your indicator using history methods in SymbolData class, in that way, we also benefit from registering the OHLC, the attribute Update will be redundant and can be removed.
• You don't need to use self.AddCrypto and self.AddUniverseSelection together, they do the same job in here. You can replace them with a single line 

symbols = [self.AddCrypto(symbol, Resolution.Minute).Symbol for symbol in ["BTCUSD","ETHUSD","LTCUSD"]]

• You will need to use .Current.Value to access the latest value in your indicators.
   

We also helped you do some simplification, hope it helps you!

Best,
Louis Szeto

Thanks Louis, greatly appreciated!

Hello Mr. Louis Szeto,

I am trying to add a ‘Close’ rolling window to an ML algo (attached) using the minute resolution consolidation to daily data method that you suggested previously for an algo that had indicators. Do you mind taking a look when you have a moment and show me how to do it for a rolling window pls? I made an attempt but not sure if this is the right way to get the rolling window to use the consolidated data.

Thanks.

Sheikh

Hi Sheikh

It is glad to see you getting more familiar with the platform and we appreciate your sharing. Just a few adjustments:

• we suggest you warm up your rolling window so that you can use it immediately,

self.Rolling = RollingWindow[float](self.lookback)
history = algorithm.History(symbol, self.lookback, Resolution.Daily)
if not history.empty:
    for index, bar in history.loc[symbol].iterrows():
        self.Rolling.Add(bar.close)

• reverse the order of rolling window data by [::-1] in data frame, as the rolling window is a queue structure that stack in at its head, not tail
• remove the DailyBarHandler method since it is no longer called,
• we select data frame by condition like this: self.df['Signal'][self.df["Return"] < 0] = -1, then you can safely remove all checking inf conditions and try-catch.

Best,
Louis Szeto

Thanks very much, Louis! Always appreciated.

I am getting a slew of 'insufficient buying power errors. I reviewed this post but not sure how to apply the solution. I tried, but no luck resolving:

if not all([algorithm.IsMarketOpen(x[symbol]) for x in symbolData.Close_rolling]):
   return []

Full code below. Kindly have a look when you get a chance. Thanks again.

import numpy as np
import pandas as pd
from sklearn.linear_model import RidgeClassifier
from AlgorithmImports import *
class MachineLearningAlgo(QCAlgorithm):
    
    def Initialize(self):
        
        self.SetStartDate(2016, 6, 2)  
        self.SetEndDate(2021, 7, 20)  
        self.SetCash(10000000)  
        
        self.SetBrokerageModel(BrokerageName.AlphaStreams)
        self.SetExecution(ImmediateExecutionModel())
        self.SetPortfolioConstruction(EqualWeightingPortfolioConstructionModel())
        self.lookback = 30
        
        symbol_list = ["BTCUSD","ETHUSD","LTCUSD","BALUSD","DAIUSD","KNCUSD",
                        "OXTUSD","RENUSD","UMAUSD","XRPUSD","ZRXUSD"]
        self.symbols = [self.AddCrypto(symbol, Resolution.Minute, Market.GDAX).Symbol for symbol in symbol_list]
        
        self.SetBenchmark("BTCUSD")
        self.AddAlpha(MachineLearningAlphaModel(self.Time, self.lookback))
                        
class MachineLearningAlphaModel(AlphaModel):
                        
    def __init__(self, Time, lookback):
        self.dataBySymbol = {}
        self.rebalanceTime = Time
        self.lookback = lookback
            
    def GetMLModel(self):
        self.MLModel = RidgeClassifier(random_state=18)   
        
    def Update(self, algorithm, data):
        insights = []
         
        if algorithm.Time < self.rebalanceTime: return []
        
        for symbol, symbolData in self.dataBySymbol.items():
            if data.Bars.ContainsKey(symbol) and not algorithm.IsWarmingUp:
                symbolData.Update(data, symbol)
                
                if symbolData.Close_rolling.IsReady:
                   
                    self.df = pd.DataFrame(symbolData.Close_rolling, columns=["Close"])[::-1].reset_index(drop=True)
                    
                    # calculate daily forward returns to be used to set Target / Signal
                    self.df['Return'] = np.log(self.df["Close"].shift(-1)/self.df["Close"]) 
                    self.df = self.df.dropna()
                    
                    # set Signal / Target
                    self.df["Signal"] = 0
                    self.df['Signal'][self.df["Return"] > 0] = 1
                    self.df['Signal'][self.df["Return"] < 0] = -1
                    
                    # set training data
                    self.X = self.df.drop(["Return", "Signal"], axis=1)
                    self.Y = self.df['Signal']
                    
                    # align feature set & signal 
                    self.Y, self.X = self.Y.align(self.X, axis=0, join='inner')
                    
                    self.X_train = self.X[:-1]
                    self.Y_train = self.Y[:-1]
                    if self.X_train.empty or self.Y_train.empty: return []
                    
                    # fit / train ML model
                    self.GetMLModel()
                    self.MLModel.fit(self.X_train, self.Y_train)
                    
                    # predict next day signal using today's values of feature set
                    self.X_today = self.X.iloc[-1]
                    # self.X_today is Series, so convert to numpy array
                    self.X_today = self.X_today.to_numpy()
                    # reshape self.X_today because it only has 1 day's sample
                    self.X_today = self.X_today.reshape(1,-1)
                    
                    # Y_predict will take predicted signal
                    self.Y_predict = self.MLModel.predict(self.X_today)
                    
                    # set insight based on predicted signal
                    # We cannot short crypto in GDAX
                    if not all([algorithm.IsMarketOpen(x[symbol]) for x in symbolData.Close_rolling]):
                        return []
                    direction = InsightDirection.Up if self.Y_predict == 1 else InsightDirection.Flat
                
        self.rebalanceTime = Expiry.EndOfDay(algorithm.Time)
                
        return insights
    
    def OnSecuritiesChanged(self, algorithm, changes):
        for change in changes.AddedSecurities:
            self.dataBySymbol[change.Symbol] = SymbolData(algorithm, change.Symbol, self.lookback)
            
        for change in changes.RemovedSecurities:
            if change.Symbol in self.dataBySymbol:
                del self.dataBySymbol[change.Symbol]
        
class SymbolData:
    def __init__(self, algorithm, symbol, lookback):
        
        self.lookback = lookback
        algorithm.Consolidate(symbol, Resolution.Daily, lambda x: None)
        
        self.Close_rolling = RollingWindow[float](self.lookback)
        
        history = algorithm.History(symbol, self.lookback, Resolution.Daily)
        if not history.empty:
            for index, bar in history.loc[symbol].iterrows():
                self.Close_rolling.Add(bar.close)
        
    def Update(self, data, symbol):
            self.Close_rolling.Add(data[symbol].Close)
+ Expand
- Collapse

Sorry, attached is the code / algo. Thanks.

Guys, 

Thanks for sharing! I learned something from it.

I have some questions regarding this advice:

Also, we would suggest using trade bar data: data.Bars.ContainsKey("SPY") and data.Bars["SPY"].Close in the original backtest to avoid error using only data["SPY"] if no bar data received. Please check docs for details.

When would such error (no bar data is received) happen? 

In the doc they look the same, but I guess maybe in live trade transmission error might be the problem?

Thanks

Hi Sheikh, Ruming Jiang,
Sheikh, to resolve the issue we recommend using the cash buffer of the algorithm. Add the following code snippet in the initialize method and refer to this backtest.
 

self.Settings.FreePortfolioValuePercentage = 0.1

Ruming Jiang,,Sometimes, the current slice object contains data about a symbol, but it is non-trade(Splits, Dividends, Delistings) data instead of a trade bar or quote bar. Refer to the attached backtest, for example.
Best,
Varad Kabade