Hi, I've been receiving logs like this in my backtests: Data for symbol METX has been limited due to numerical precision issues in the factor file. The starting date has been set to 3/30/2020. This appears for nearly all the securities in the selection.
My agorithm is basically a copy from the Micro-cap growth strategy from What Works on Wall Street. It filters first by market cap, then by Price-to-Sales Ratio, then arranges the selection in descending order of one-year Rate-of-Return (not the exact formula but it does the job) of the stock. I think the problem lies in a lack of fundamental data, and I would like to exclude all securities that do not have fundamental data updated in the last 3 months.
I think someone has already mentioned this issue before in a different context:
https://www.quantconnect.com/forum/discussion/7682/remove-securities-that-don-039-t-have-current-data/p1However, the solution given was to just wait till the next data point arrives. Jusging from the starting dates that was set for my selection, this would probably take months which would be an impractical waiting time, so I was wondering if someone has a workaround to suggest. Thanks a lot! Below is the code.
P.S. in the final state of things, "TOPS" somehow had an adjusted price increase by more than 2970 times, and I'm not sure how that even happened :p I'm just ignoring that for now.
from QuantConnect.Data.UniverseSelection import *
import math
import numpy as np
import pandas as pd
import scipy as sp
class MicroGrowth(QCAlgorithm):
def Initialize(self):
#self.SetStartDate(2020, 2, 12) # Set Start Date
self.SetStartDate(2020, 2, 28)
self.SetEndDate(2021, 3, 1)
self.SetCash(100000) # Set Strategy Cash
self.UniverseSettings.Resolution = Resolution.Daily
self.AddUniverse(self.CoarseSelectionFunction, self.FineSelectionFunction)
self.lastmonth = -1
self.monthinterval = 3
self.Symbols = None
def OnData(self, data):
'''OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
Arguments:
data: Slice object keyed by symbol containing the stock data
'''
def CoarseSelectionFunction(self,coarse):
if self.lastmonth == -1 or ((self.Time.month-self.lastmonth) % self.monthinterval == 0 and self.Time.month != self.lastmonth):
self.lastmonth = self.Time.month
return [x.Symbol for x in coarse if x.HasFundamentalData]
else:
return Universe.Unchanged
def FineSelectionFunction(self,fine):
#momo_dict = {}
momo_list = []
MKTCAP_dict = {}
#exclude delisted
excluded_delisted = [i for i in fine if isinstance(i.SecurityReference.DelistingDate.date(),datetime) == False]
#filter by mkt_cap
for i in fine:
if isinstance(i.MarketCap,(float,int)) and i.MarketCap != 0:
MKTCAP_dict[i]=i.MarketCap
microcap = [i for i in excluded_delisted if isinstance(MKTCAP_dict.get(i),(int,float)) and MKTCAP_dict.get(i)>25e6 and MKTCAP_dict.get(i)<250e6]
#filter by Price-to-Sales Ratio < 1 (defined to be null if result <= 0)
micro_PSR = [i for i in microcap if isinstance(i.ValuationRatios.PSRatio,(float,int)) and i.ValuationRatios.PSRatio < 1 and i.ValuationRatios.PSRatio > 0]
#sorting by momentum
hist = self.History([i.Symbol for i in micro_PSR], 365, Resolution.Daily)
for i in micro_PSR:
curr_price = hist.loc[str(i.Symbol)]['close'][-1]
baseline_price = hist.loc[str(i.Symbol)]['close'][0]
if i.Symbol.Value == "TOPS":
self.Log(f'CURRENT PRICE: {curr_price}, BASELINE PRICE: {baseline_price}')
momo = curr_price/baseline_price
momo_list.append(momo)
#momo_dict[i] = momo
micro_PSR_symbols = [f.Symbol.Value for f in micro_PSR]
MKTCAP_list = ["{:e}".format(f.MarketCap) for f in micro_PSR]
PSR_list = [f.ValuationRatios.PSRatio for f in micro_PSR]
momo_zip = list(zip(micro_PSR, micro_PSR_symbols, MKTCAP_list, PSR_list, momo_list))
micro_PSR_momo = sorted(momo_zip,key = lambda i: i[-1], reverse = True)
output = [f[0].Symbol for f in micro_PSR_momo[:50]]
self.Symbols = [f[1] for f in micro_PSR_momo[:50]]
self.Log(f'CURRENT STATE: {[f[1:] for f in micro_PSR_momo[:50]]}')
return output
def OnSecuritiesChanged(self, changes):
# selected symbols will be found in Log
self.Log(f'New Securities Added: {[security.Symbol.Value for security in changes.AddedSecurities]}')
self.Log(f'Securities Removed{[security.Symbol.Value for security in changes.RemovedSecurities]}')
#self.Log(f'CURRENT STATE: {self.Symbols}')
Derek Melchin
Hi Han,
The "numerical precision" issue occurs because some factor files have INF split values, indicating that the stock has so many splits that prices can't be calculated with correct numerical precision. To allow backtesting these symbols, we need to move the starting date forward when reading the data. We have an open GitHub Issue to reduce the number of times this error message is displayed. Subscribe to our progress here.
TOPS shows a 2970x return because we are missing a split event in our records. The attached notebook shows the price of TOPS jumps from $0.003 to $1.719 in 1 day. We are working to resolve this issue. Subscribe to our progress here.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Han Ruobin
Thank you for your reply! Moving the starting date forward would be defeating the purpose of running the algo since I require the most updated data (at least within the past 3 months). Is there a way to filter out the securities with an INF split value (such as by comparing an attribute or something)? Thank you!
Derek Melchin
Hi Han,
The coarse-fine universe selection model already filters out securities with an INF split value. For instance, the logs of the algorithm above shows
Data for symbol ALYA has been limited due to numerical precision issues in the factor file. The starting date has been set to 11/1/2018.
However, note that the algorithm above has a start date of 28/2/2020. In the algorithm attached below, where we start the backtest before 11/1/2018, we see ALYA is not added to the universe until after 11/1/2018.
We've amended the GitHub Issue related to this error message to only display the error when the start date of the backtest is before the date displayed in the error (11/1/2018 for ALYA).
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Han Ruobin
Hi Derek,
Thanks for your reply! Can I take it that the "numerical precision" issues will not occur for all dates after the "starting date" (e.g. 11/1/2018 for ALYA)?
Derek Melchin
Hi Han,
Yes, that's correct.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Han Ruobin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!