Share New Research 
I have a simple custom data as follows. History calls on the custom data are 10x slower both in the research environment and during backtests, compared to history calls on native Crypto datatypes. How do I make it faster?
class CryptoAuxData(PythonData):
def GetSource(self, config, date, isLiveMode):
return SubscriptionDataSource(
os.path.join(Globals.DataFolder, f"crypto/binance/hour_aux/{config.Symbol.Value.lower()}.csv"),
SubscriptionTransportMedium.LocalFile
)
def Reader(self, config, line, date, isLiveMode):
coin = CryptoAggData()
coin.Symbol = config.Symbol
try:
data = line.split(',')
value = float(data[4])
if value <= 0: return None
coin.Time = datetime.strptime(data[0], "%Y%m%d %H:%M")
coin.Value = value
# coin.EndTime = coin.Time + timedelta(minutes=5)
coin["Open"] = float(data[1])
coin["High"] = float(data[2])
coin["Low"] = float(data[3])
coin["Close"] = float(data[4])
coin["Volume"] = float(data[5])
coin["NumTrades"] = float(data[6])
coin["MarketBuy"] = float(data[7])
return coin
except ValueError:
# Do nothing, possible error in json decoding
return None
Mia Alissi
The performance difference you're experiencing between history calls on custom data and native Crypto datatypes is expected. Native QC data gets the full benefit of security modeling like splits and dividends, and is optimized for performance. On the other hand, custom data is ignorant of everything about the actual market, and things like market hours, delistings, and mergers don't work with custom data. This can lead to slower performance when dealing with custom data.
However, there are a few things you can do to potentially improve the performance of your custom data:
Data Resolution: The resolution of your custom dataset should match the actual resolution of your data. The lowest reasonable resolution is every minute. Anything more frequent than every minute is very slow to execute. The frequency that LEAN checks the data source depends on the resolution argument.
Data Parsing: Ensure that your
Readermethod is as efficient as possible. This method is called for each line of your data, so any performance improvements here can have a big impact. In your case, you might want to consider using a faster method for parsing your CSV data, or handling errors in a more efficient way.Data Access: As your data reader reads your custom data file, LEAN adds the data points in the
Sliceit passes to your algorithm'sOnDatamethod. To collect the custom data, use theSymbolor name of your custom data subscription. You can access theValueand custom properties of your custom data class from theSlice. To access the custom properties, use the custom attribute pass the property name to theGetPropertymethod.Remember, your
Readermethod should return objects in chronological order. If an object has a timestamp that is the same or earlier than the timestamp of the previous object, LEAN ignores it.For more information, you can refer to the following documentation:
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Bhuang
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!