Hey Everyone!
Next up in our series is an algorithm that shows how to find uncorrelated assets. This allows you to find a portfolio that will, theoretically, be more diversified and resilient to extreme market events. When combined with other indicators and data sources, this can be an important component in building an algorithm that limits drawdown and remains profitable in choppy markets.
The first step is to experiment with the code we'll need in the research notebook. We don't need to pick any specific tickers here, just enough to make sure our code works and that everything will transfer seamlessly when dropped into an algorithm. In the notebook, we grab the historical data for the securities and use this to fetch returns data. Then, we calculate the correlation of the returns, which gives us a correlation matrix. The last step is to figure out which symbols have the lowest overall correlation with the rest of the symbols as a whole -- we want to find the five assets with the lowest average absolute correlation so that we can trade them without fearing that any pair are too highly correlated.
# Write our custom function
def GetUncorrelatedAssets(returns, num_assets):
# Get correlation
correlation = returns.corr()
# Find assets with lowest mean correlation, scaled by STD
selected = []
for index, row in correlation.iteritems():
corr_rank = row.abs().mean()/row.abs().std()
selected.append((index, corr_rank))
# Sort and take the top num_assets
selected = sorted(selected, key = lambda x: x[1])[:num_assets]
return selected
## Perform operations
qb = QuantBook()
tickers = ["SQQQ", "TQQQ", "TVIX", "VIXY", "SPLV",
"SVXY", "UVXY", "EEMV", "EFAV", "USMV"]
symbols = [qb.AddEquity(x, Resolution.Minute) for x in tickers]
# Fetch history
history = qb.History(qb.Securities.Keys, 150, Resolution.Hour)
# Get hourly returns
returns = history.unstack(level = 1).close.transpose().pct_change().dropna()
# Get 5 assets with least overall correlation
selected = GetUncorrelatedAssets(returns, 5)
With this code, we can perform the same operation in an algorithm. To demonstrate this, we've built an algorithm that uses basic Universe Selection and Scheduled Events. The algorithm is initialized using familiar models -- Equal Weighting Portfolio Construction, Immediate Execution, and a Scheduled Event set to rebalance our holdings every day 5-minutes after market open.
def Initialize(self):
self.SetStartDate(2019, 1, 1) # Set Start Date
self.SetCash(1000000) # Set Strategy Cash
self.UniverseSettings.Resolution = Resolution.Minute
self.AddUniverse(self.CoarseSelectionFunction)
self.SetBrokerageModel(AlphaStreamsBrokerageModel())
self.SetPortfolioConstruction(EqualWeightingPortfolioConstructionModel())
self.SetExecution(ImmediateExecutionModel())
self.AddEquity('SPY')
self.SetBenchmark('SPY')
self.Schedule.On(self.DateRules.EveryDay('SPY'), self.TimeRules.AfterMarketOpen("SPY", 5), self.Recalibrate)
self.symbols = []
The universe is selected using the top 100 symbols by traded dollar volume.
def CoarseSelectionFunction(self, coarse):
sortedByDollarVolume = sorted(coarse, key=lambda x: x.DollarVolume, reverse=True)
filtered = [ x.Symbol for x in sortedByDollarVolume ][:100]
return filtered
Then we copy and paste our research notebook code into the OnSecuritiesChanged method, which allows us to perform our filtering on new symbols in the universe (technically we haven't done an exact copy-and-paste this time as we had to change the symbol argument being passed to qb.History). Additionally, we want to emit Flat Insights for any symbols that have been removed from the Universe so that we ensure our positions are closed.
def OnSecuritiesChanged(self, changes):
symbols = [x.Symbol for x in changes.AddedSecurities]
qb = self
# Copied from research notebook
#---------------------------------------------------------------------------
# Fetch history
history = qb.History(symbols, 150, Resolution.Hour)
# Get hourly returns
returns = history.unstack(level = 1).close.transpose().pct_change().dropna()
# Get 5 assets with least overall correlation
selected = GetUncorrelatedAssets(returns, 5)
#---------------------------------------------------------------------------
# Add to symbol dictionary for use in Recalibrate
self.symbols = [symbol for symbol, corr_rank in selected]
# Emit Flat Insights for removed securities
symbols = [x.Symbol for x in changes.RemovedSecurities]
insights = [Insight.Price(symbol, timedelta(minutes = 1), InsightDirection.Flat) for symbol in symbols if self.Portfolio[symbol].Invested]
self.EmitInsights(insights)
Finally, we use the Scheduled Event to generate our Insights!
def Recalibrate(self):
insights = []
insights = [Insight.Price(symbol, timedelta(5), InsightDirection.Up, 0.03) for symbol in self.symbols]
self.EmitInsights(insights)
That's it! The main thing to take away from this is how you can quickly move between the research environment and real algorithms even when we had to change a few things between the two in this example. This provides you with a lot of power in being able to apply research practices easily in algorithms, which hopefully will help you find new sources of alpha!
(Please note: this universe is not valid for the competition. Using coarse selection was done just for the purpose of demonstration)
Tim Bohmann
Would like to see the resulting code in C# version. This is great information!
Alexandre Catarino
Hi Tim Bohmann ,
Thank you for the suggestion.
We will try to add examples in C# ASAP.
QuantConnect has white-listed the following mathematical packages:
Accord AForge Math.Net Numerics AlgoLib Math.Net FilteringAli Badr
Amazing tutorial... God bless
Anthony FJ Garner
Jack Simonson
Jack, I wonder whether there might be some misconception here? In the notebook, the correlation calculation works as expected. Out of 10 stocks you choose the 5 least uncorrelated.
That works also on the FIRST day of the algorithm when ALL 100 course filtered stocks are additions in OnSecuritiesChanged.
On subsequent days however you will NOT be calculating the correlation on the entire potential portfolio of 100 stocks (5 of which you will now be invested in) but only on the ADDED stocks.
See line 41:
symbols = [x.Symbol for x in changes.AddedSecurities]
It is these symbols for which you download history and for which you assess correlations.
I think perhaps the correlation function needs to be moved to the CourseSelection function. Or elsewhere. Forgive me if I have misunderstood this algorithm - I am very new to Quantconnect and struggling!
Alexandre Catarino
Hi Anthony FJ Garner ,
You have a good point. Instead of using AddedSecurities, the algorithm could use active securities (minus the RemovedSecurities):
symbols = [x.Symbol for x in self.ActiveSecurities.Values if x.Symbol not in changes.RemovedSecurities]
I don't think we can say that this is conceptually better or worse than the previous version, but I am leaving the alternative for those who want to check it out.
Mark Reeve
Hi Alexandre Catarino,
Why do you subtrract the RemovedSecurities?
Wouldnt you want the entire ActiveSecurities universe when comparing correlations?
Thanks,
Mark
Mark Reeve
... And continuing on from Anthony FJ Garner's comment, again your correlation code will only fire when their is a change in to one of the securities in your universe as we are only comparing correlation within:
def OnSecuritiesChanged(self, changes):
I believe we should be re-analysing the 150 hour correlations for our universe EVERYDAY before we place our trades, and therefore should be requesting history and analysing correlations within Recalibrate(), something like this:
def Recalibrate(self): insights = [] symbols = [x.Symbol for x in self.ActiveSecurities.Values] qb = self history = qb.History(symbols, 150, Resolution.Hour) returns = history.unstack(level = 1).close.transpose().pct_change().dropna() selected = GetUncorrelatedAssets(returns, 5) self.symbols = [symbol for symbol, corr_rank in selected] insights = [Insight.Price(symbol, timedelta(5), InsightDirection.Up, 0.03) for symbol in self.symbols] self.EmitInsights(insights)
Derek Melchin
Hi Mark,
Both great points.
By moving the correlation analysis and insight generation into the Recalibrate method, we ensure we are invested in the 5 most uncorrelated assets at each rebalance instead of after each universe change. However, we must remember to emit flat insights for the symbols we are invested in but aren't in `self.symbols`.
Additionally, ActiveSecurities won't include any of the RemovedSecurities after moving the code into Recalibrate, so the condition is not necessary.
See the attached algorithm for an implementation of these changes.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Zach Oakes
I'm pretty sure there's a bug on line 12 in notebook example
# Get hourly returns returns = history.unstack(level = 1).close.transpose().pct_change().dropna()
-- should be either calculating pct_change() first, THEN .transpose() (get pct change of symbols in columns)
OR
using pct_change(axis='columns') after the transpose -- otherwise you're calculating pct of the columns when the symbols are now in the rows. Basically the nan value will be the first security, and you drop it.
Arthur Asenheimer
Hi Zach,
I'm pretty sure there is no bug on line 12. You might have overlooked that history is a multi-index dataframe and level = 1 doesn't contain the symbol-index but the time-index. So we still have symbols as rows but time as columns after unstacking . That's why we need to transpose the dataframe after unstacking but before we calculate the percentage changes. Check out the research.ipynb file in the attached backtest.
It's easier to understand when you apply the commands step by step which is what I did in the attached research.
Zach Oakes
Arthur, Yeah I overlooked that symbols were row index, I was picturing it as Column index (like pandas_datareader).
Zach Oakes
I'm getting some weird behavior with this -- I'm getting a random index out of bounds error with the unstack call (in system -- not in Research) -- and for some reason this first line after the loop begins isn't working (not matching the symbols from uncorr return -- but does match for the basic symbols call if uncorr is turned off.
Zach Oakes
def OnSecuritiesChanged (self, changes): symbols = [x.Symbol for x in changes.AddedSecurities] #Returns <GOOG SYMBOLID> if self.uncorr and len(symbols) > 0: top = self.top_x #int(self.top_x / 2) if self.mkt_cap_sort else self.top_x history = self.History(symbols, 150, Resolution.Hour) if history.shape[1] > 1: hist = history.unstack(level = 1).close.transpose().pct_change().dropna() #WHY does this get out of index error ^^ symbols_rank = GetUncorrelatedAssets(hist, top) symbols = [symbol for symbol, corr_rank in symbols_rank] #for s, s2 in zip(symbols, symbols_new): # self.Debug(f'{s} - {s2}') #Identical? WHY 168 not working? for x in changes.AddedSecurities: if x.Symbol not in symbols: continue
Shile Wen
Hi Zach,
Please attach the full algorithm to help us identify the problem. For now, our debugger is a great resource to see if there are any bugs in the logic.
Best Regards,
Shile Wen
Zach Oakes
I have tried running the little GetUncorrelated loop in Fine, and in OnSecuritesChanged
Zach Oakes
My questions (In OnSecuritiesChanged) --
why I need this line to confirm history worked: (Otherwise getting an out of index error w/in the GetUncorr function):
if history.shape[1] > 1:
and why this is inconsisent:
if x.Symbol not in symbols: continue
This ^^ line does match when it uses
symbols = [x.Symbol for x in changes.AddedSecurities]
does NOT match with:
symbols_rank = GetUncorrelatedAssets(hist, top)
s2 = [symbol for symbol, corr_rank in symbols_rank]
symbols = [s for s in symbols if s in s2]
#Or Any other variations I've tried..
Derek Melchin
Hi Zach,
The out of index error occurs when we try unstacking an empty DataFrame. Having
if history.shape[1] > 1:
prevents that from occurring.
The symbols in
[x.Symbol for x in changes.AddedSecurities]
won't match those returned from `GetUncorrelatedAssets` because `GetUncorrelatedAssets` returns a list of strings, not Symbols. Also, the length of the lists will be different if the `num_assets` argument we provide to `GetUncorrelatedAssets` is less than the length of `changes.AddedSecurities`.
To convert a symbol in string format to a symbol object, use the Symbol method.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Zach Oakes
Thanks... I can't find any examples of just Symbol being used (aside from in like init) -- can you give me a conversion example? It looks like it just takes a string... not much to it -- prly requires the algo instance.
syms = [self.Symbol(sym_string) for sym_string, rank in symbols_rank if self.Symbol(sym_string) in changes.AddedSecurities] #I also tried this, shouldn't this work even if it is a string, as x.Symbol should be a string, no? s4 = [x.Symbol for x in changes.AddedSecurities if x.Symbol in symbols_rank]
Shile Wen
Hi Zach,
If we add a stock, say AAPL, we can get the Symbol object with self.Symbol('AAPL').
Please see the attached backtest for reference.
Best,
Shile Wen
Jack Simonson
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!