Share New Research 
Hi! I have a simple liquid universe algo that filters fine data, but now I'm trying to add a momentum filter(EMAs) before having the algo filter by fine data for the liquid universe. I tried to incorporate the code from this bootcamp into my existing algo, but I'm encountering errors. I'm trying to find what I'm missing, but I can't figure it out.
I know the code is wrong because I removed the Universe Selection in Initialize; I only did that so a backtest could be produced so I could attach the algorithm to this post. If I left in the Universe Selection in Initialize it gives me the following error:
Runtime Error: AttributeError : 'TechnologyUniverseModule' object has no attribute 'History'
at <lambda> in FundamentalUniverseSelectionModel.py:59
at FilteredSelectCoarse in FundamentalUniverseSelectionModel.py:72
AttributeError : 'TechnologyUniverseModule' object has no attribute 'History' (Open Stacktrace)
Vncne
The coarse selection part was done like this:
def __init__(self, filterFineData = True, universeSettings = None, securityInitializer = None): #Initializes a new default instance of the TechnologyUniverseModule super().__init__(filterFineData, universeSettings, securityInitializer) self.numberOfSymbolsCoarse = 1000 self.numberOfSymbolsFine = 100 self.dollarVolumeBySymbol = {} self.lastMonth = -1 self.averages = {} def SelectCoarse(self, algorithm, coarse): if algorithm.Time.month == self.lastMonth: return Universe.Unchanged coarse = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10], key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] for security in coarse: symbol = security.Symbol if symbol not in self.averages: history = self.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(self.Time, security.AdjustedPrice) if self.averages[symbol].is_ready() and self.averages[symbol].fast > self.averages[symbol].slow: self.dollarVolumeBySymbol.append(symbol) if len(self.dollarVolumeBySymbol) == 0: return Universe.Unchanged return list(self.dollarVolumeBySymbol.keys())and for constructing the indicators:
class SelectionData(): def __init__(self, history): self.slow = ExponentialMovingAverage(200) self.fast = ExponentialMovingAverage(50) for bar in history.itertuples(): self.fast.Update(bar.Index[1], bar.close) self.slow.Update(bar.Index[1], bar.close) def is_ready(self): return self.slow.IsReady and self.fast.IsReady def update(self, time, price): self.fast.Update(time, price) self.slow.Update(time, price)Shile Wen
Hi Vncne,
To use History in a Framework component, we need to call algorithm.History instead of self.History. In addition, I also fixed one instance where self.Time was used instead of algorithm.Time. Furthermore, the algorithm uses append on a dict, so I’ve also fixed this as well. Please see the attached backtest for reference.
Best,
Shile Wen
Vncne
You're the best Shile Wen , thanks! Your code fixed it, but I wanted to move the indicator filter to be done after the fine filter instead of right after the coarse filter and before the fine filter originally so the algorithm has less data it needs to filter through.
But I now get this error:
Runtime Error: AttributeError : 'FineFundamental' object has no attribute 'AdjustedPrice'
at <lambda> in FundamentalUniverseSelectionModel.py:46
AttributeError : 'FineFundamental' object has no attribute 'AdjustedPrice' (Open Stacktrace)
By the way, in the algorithm I attached to this reply, I removed the self.AddUniverseSelection() in Initialize in order for the algorithm to produce a backtest that I would be able to attach to this post;
#self.AddUniverseSelection(TechnologyUniverseModule()) self.AddRiskManagement(NullRiskManagementModel()) self.SetPortfolioConstruction(NullPortfolioConstructionModel()) self.SetExecution(ImmediateExecutionModel())Thanks again, you're a huge help!
Vncne
Sorry, just another question; I also noticed that these produce different results:
coarse = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10], key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] for security in coarse: symbol = security.Symbol self.dollarVolumeBySymbol[symbol] = security.DollarVolume return list(self.dollarVolumeBySymbol.keys())and:
coarse = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10], key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in coarse} return list(self.dollarVolumeBySymbol.keys())Why do the outputs of both of these differ?
Arthur Asenheimer
Hi,
in your first version, you're adding items to your dollarVolumeBySymbol dictionary without any refresh/cleanup. So the dictionary will keep all the keys from your past Coarse Selection calls.
In your second version, you're creating a new dictionary every time Coarse Selection is invoked.
To the question of your previous post (AttributeError: FineFundamental object has no attribute AdjustedPrice):
As the error message suggests, AdjustedPrice is not available in FineFundamental. You can see here that AdjustedPrice is not among the properties of the FineFundamental class.
This was something I also needed some time ago. I then decided to move some parts of my code to the CoarseSelection method but that wasn't a very elegant solution, to be honest.
Maybe Shile can offer a solution or workaround here.
Vncne
Hi Arthur Asenheimer , thanks for the response! Yeah I figured as such with the first version(in regards to not creating a new dict every time) So I guess this implies that there are times where symbols that should be removed from the universe are not liquidated? Maybe I'm misunderstanding again.
How would you suggest I incorporate this:
self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in self.averages}into this code, in order to make sure that the universe is always refreshed and up to date?
for security in sortedByDollarVolume: symbol = security.Symbol if symbol not in self.averages: history = algorithm.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(algorithm.Time, security.AdjustedPrice) if self.averages[symbol].is_ready(): self.dollarVolumeBySymbol[symbol] = security.DollarVolumeVncne
What I tried:
def SelectCoarse(self, algorithm, coarse): selected = [] if algorithm.Time.month == self.lastMonth: return Universe.Unchanged sortedByDollarVolume = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10], key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] for security in sortedByDollarVolume: symbol = security.Symbol if symbol not in self.averages: history = algorithm.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(algorithm.Time, security.AdjustedPrice) if self.averages[symbol].is_ready(): selected[symbol] = security.DollarVolume self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in selected} # If no security has met the QC500 criteria, the universe is unchanged. if len(self.dollarVolumeBySymbol) == 0: return Universe.Unchanged return list(self.dollarVolumeBySymbol.keys())Error:
Runtime Error: TypeError : list indices must be integers or slices, not Symbol
at <lambda> in FundamentalUniverseSelectionModel.py:59
at FilteredSelectCoarse in FundamentalUniverseSelectionModel.py:72
TypeError : list indices must be integers or slices, not Symbol (Open Stacktrace)
Vncne
Oh yeah, just to clarify regarding what you said about the first version not being refreshed/cleaned up; is this still true even if I added this function before the coarse function?
class TechnologyUniverseModule(FundamentalUniverseSelectionModel): def __init__(self, filterFineData = True, universeSettings = None, securityInitializer = None): self.numberOfSymbolsCoarse = 1000 self.numberOfSymbolsFine = 100 self.dollarVolumeBySymbol = {} self.lastMonth = -1 self.averages = {} super().__init__(filterFineData, universeSettings, securityInitializer) def SelectCoarse(self, algorithm, coarse):Would self.dollarVolumeBySymbol = {} technically refresh the dict/list every time before proceeding to the coarse and fine filters?
Arthur Asenheimer
The __init__ method will be called only once at the beginning when creating an instance of the class TechnologyUniverseModule. So this will not cause a refresh every time before proceeding the coarse/fine selection.
Re your question to pick vers 1 or vers 2, there is no right answer. But in most cases it's more accurate to refresh your list/dict. So vers 2 is more appropriate here.
In your CoarseSelection method you can do:
self.dollarVolumeBySymbol = { .... some dict comprehension logic ... }or
self.dollarVolumeBySymbol = {}for symbol in mylist:
self.dollarVolumeBySymbol[symbol] = value
I recommend you to learn the OOP principles and python basics. There is plenty of good free ressources online.
Arthur Asenheimer
And one more thing: the debugger can be very helpful in such situations and let you find out a lot stuff yourself.
Vncne
I'm actually working on a Udemy course now! But I'm also trying to learn QC api mostly through trial and error, so I apologize for all the basic questions;
in CoarseSelection method I tried:
for security in sortedByDollarVolume: symbol = security.Symbol if symbol not in self.averages: history = algorithm.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(algorithm.Time, security.AdjustedPrice) if self.averages[symbol].is_ready(): self.dollarVolumeBySymbol[symbol] = security.DollarVolume self.dollarVolumeBySymbol = {x.Symbol:x.DollarVolume for x in self.dollarVolumeBySymbol \ if self.averages[x.Symbol].is_ready()}But it returns an error:
Runtime Error: AttributeError : 'Symbol' object has no attribute 'Symbol'
at <lambda> in FundamentalUniverseSelectionModel.py:59
at FilteredSelectCoarse in FundamentalUniverseSelectionModel.py:72
AttributeError : 'Symbol' object has no attribute 'Symbol' (Open Stacktrace)
I apologize again and I appreciate the patience of everyone on QC for the beginners, and thanks again!
Arthur Asenheimer
The error message desribes it quite well: "Symbol" object has no attribute "Symbol".
You're trying to do x.Symbol where x is already the Symbol object. Just remove the ".Symbol" part in the corresponding line and it should work then. I guess the error is thrown in the line
self.dollarVolumeBySymbol = {x.Symbol : x.DollarVolume for x in self.dollarVolumeBySymbol if self.averages[x.Symbol].is_ready()}Replace this line with:
self.dollarVolumeBySymbol = { symbol : dollarVolume for symbol, dollarVolume in self.dollarVolumeBySymbol.items() if self.averages[symbol].is_ready() }Vncne
Ah okay, I assumed that the symbol = security.Symbol was confined within the for loop that I put it in. I tried following your suggestion and implemented it like this:
def SelectCoarse(self, algorithm, coarse): selected = {} if algorithm.Time.month == self.lastMonth: return Universe.Unchanged sortedByDollarVolume = sorted([x for x in coarse if x.HasFundamentalData and x.Price > 10], key = lambda x: x.DollarVolume, reverse=True)[:self.numberOfSymbolsCoarse] for security in sortedByDollarVolume: symbol = security.Symbol if symbol not in self.averages: history = algorithm.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(algorithm.Time, security.AdjustedPrice) selected[symbol] = security.DollarVolume self.dollarVolumeBySymbol = {symbol:dollarVolume for symbol, dollarVolume in selected.items() if self.averages[symbol].is_ready()} if len(self.dollarVolumeBySymbol) == 0: return Universe.Unchanged return list(self.dollarVolumeBySymbol.keys())It seems to be working, but does that look right to you?
Vncne
Hi Shile Wen, after following your and Arthur's suggestions, I was more or less able to run backtests on the algorithm with relatively no more issues. However, when deploying the algorithm live, I've been encountering a different issue. Everyday at around 11:30, the live deployment has been producing the error below most likely due to an overload of history requests:
Error:
Runtime Error: Algorithm took longer than 10 minutes on a single time loop. CurrentTimeStepElapsed: 10.0 minutes
Most likely from this code:
for security in sortedByDollarVolume: symbol = security.Symbol if symbol not in self.averages: history = algorithm.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(algorithm.Time, security.AdjustedPrice) selected[symbol] = security.DollarVolumeToo many history requests. This is within the CoarseFilter function. An obvious solution would be to move the history request to the FineFilter function so that there is a significantly lower number of securities to request history for, but like Arthur said above, AdjustedPrice is not available in FineFundamental. So, moving the history request to FineFilter produces another different error:
Runtime Error: AttributeError : 'FineFundamental' object has no attribute 'AdjustedPrice'
at <lambda> in FundamentalUniverseSelectionModel.py:46
AttributeError : 'FineFundamental' object has no attribute 'AdjustedPrice' (Open Stacktrace)
A suggestion I received from someone else was to save AdjustedPrice from Coarse in a class variable dictionary so that it may also be used in FineFilter. I'm not exactly sure how to do that, but I tried it like this; I saved it in Coarse like:
self.AdjustedPrice = {x.Symbol:x.AdjustedPrice for x in sortedByDollarVolume}Then attempted to use it in Fine like:
for security in sortedByDollarVolume: symbol = security.Symbol if symbol not in self.averages: history = algorithm.History(symbol, 200, Resolution.Daily) self.averages[symbol] = SelectionData(history) self.averages[symbol].update(algorithm.Time, self.AdjustedPrice) selected[symbol] = security.DollarVolumeBut now end up with this error:
Runtime Error: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the Update method. Please checkout the API documentation.
at <lambda> in FundamentalUniverseSelectionModel.py:46
TypeError : No method matches given arguments for Update (Open Stacktrace)
Derek Melchin
Hi Vncne,
The error in the code above surfaces because we are passing the `AdjustedPrice` dictionary to the update method when it's expecting a float. To resolve this, we can define SelectCoarse to include
self.last_coarse = {cf.Symbol: cf for cf in sorted_by_dollar_volume}Then in SelectFine, we can access the AdjustedPrice with
self.last_coarse[symbol].AdjustedPriceWe can then reduce the number of History requests in SelectFine to just one with
new_symbols = [] for security in sorted_by_dollar_volume: symbol = security.Symbol if symbol not in self.averages: new_symbols.append(symbol) history = algorithm.History(new_symbols, 200, Resolution.Daily) selected = [] for security in sorted_by_dollar_volume: symbol = security.Symbol if symbol in new_symbols: self.averages[symbol] = SelectionData(history.loc[symbol]) self.averages[symbol].update(algorithm.Time, self.last_coarse[symbol].AdjustedPrice) if self.averages[symbol].is_ready() and self.averages[symbol].fast > self.averages[symbol].slow: selected.append(symbol)This adjustment just requires us to change the SelectionData constructor to read
for time, row in history.iterrows(): self.fast.Update(time, row.close) self.slow.Update(time, row.close)See the attached backtest for reference.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Vncne
Thank you Derek!! With a little modification, I was able to get it working the way I wanted using your suggestion!
Vncne
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!