Custom Securities
Key Concepts
Introduction
To receive your custom data in the on_data
method instead of in a bulk download, create a custom type and then create a data subscription. The custom data type tells LEAN where to get your data and how to read it. All custom data types must extend from PythonData
and override the reader
and get_source
methods.
Unlike custom data, native QC data gets the full benefit of security modeling like splits and dividends. Custom data is ignorant of everything about the actual market. Things like market hours, delistings, and mergers don't work with custom data.
Create Subscriptions
After you define the custom data class, in the initialize
method of your algorithm, call the add_data
method. This method gives LEAN the type factory to create the objects, the name of the data, and the resolution at which to poll the data source for updates.
class MyAlgorithm(QCAlgorithm): def initialize(self) -> None: self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).symbol
The resolution
argument should match the resolution of your custom dataset. The lowest reasonable resolution is every minute. Anything more frequent than every minute is very slow to execute. The frequency that LEAN checks the data source depends on the resolution
argument. The following table shows the polling frequency of each resolution:
Resolution | Update Frequency |
---|---|
Daily | Every 30 minutes |
Hour | Every 30 minutes |
Minute | Every minute |
Second | Every second |
Tick | Constantly checks for new data |
There are several other signatures for the add_data
method.
self.add_data(type, ticker, resolution) self.add_data(type, ticker, resolution, time_zone, fill_forward, leverage) self.add_data(type, ticker, properties, exchange_hours, resolution, fill_forward, leverage)
Receive Custom Data
As your data reader reads your custom data file, LEAN adds the data points in the Slice
it passes to your algorithm's on_data
method. To collect the custom data, use the symbol
or name of your custom data subscription. You can access the value
and custom properties of your custom data class from the Slice
. To access the custom properties, pass the property name to the get_property
method.
class MyAlgorithm(QCAlgorithm): def on_data(self, slice: Slice) -> None: if slice.contains_key(self._symbol): custom_data = slice[self._symbol] value = custom_data.value property1 = custom_data.property1
Debugging
In backtests, you can use the IDE debugging capabilities to inspect the variables to understand the internal state of the custom data implementation. For more information about backtest debugging, see the Debugging documentation for Cloud Platform, Local Platform, or the CLI.
In live trading, consider adding logging statements to help with debugging since the debugger isn't available. To log information from inside your custom data class, set a class member that references the algorithm.
class MyCustomDataTypeAlgorithm(QCAlgorithm): def initialize(self): MyCustomDataType.ALGORITHM = self self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).Symbol class MyCustomDataType(PythonData): def reader(self, config: SubscriptionDataConfig, line: str, date: datetime, is_live_mode: bool) -> BaseData: if not line[0].isdigit(): # Display the line with the header MyCustomDataType.ALGORITHM.Debug(f"HEADER: {line}"); return None data = line.split(',') custom = MyCustomDataType() custom.symbol = config.symbol custom.end_time = datetime.strptime(data[0], '%Y%m%d') + timedelta(1) custom.value = float(data[1]) return custom
Avoid Look-Ahead Bias
Look-ahead bias occurs when you make decisions with information that wouldn't be available until some time in the future. In backtesting, look-ahead bias occurs when you receive data earlier than it would actually be available in reality. If look-ahead bias seeps into your algorithm, it will perform differently between live trading and backtesting.
To avoid look-ahead bias, set the timestamp of data points to the time when the data would actually be available. A lot of external sources apply timestamps to data differently than we do. For instance, on some platforms, daily bars are displayed with the date that they opened. We display daily bars with the date they closed. If you set the end_time
to the start of the bar, you'll receive the bar earlier in backtests than you would in live trading.
Time Modeling
Data types classes in LEAN inherit from the BaseData
class that defines the time
and end_time
properties. These properties represent the time period of which the data was built. If the data type occurs in a singular point in time, they have no period, so time
and end_time
are the same. Regardless of the period, LEAN uses the time when the data sample ends, end_time
, to add the sample to a Slice
.
Historical Data
To get historical data points for a custom dataset, call the history
method with the dataset Symbol
.
For more information about historical data, see Data Points.
Market Hours
The default market hours for custom securities is to be always open.
To set the market hours, pass an exchange_hours
argument to the add_data
method.
symbol = Symbol.create('SPY', SecurityType.EQUITY, Market.USA) symbol_properties = self.symbol_properties_database.get_symbol_properties(symbol.id.market, symbol, SecurityType.EQUITY, 'USD') security_exchange_hours = self.market_hours_database.get_exchange_hours(symbol.id.market, symbol, SecurityType.EQUITY) security = self.add_data(CustomType, contract, symbol_properties, security_exchange_hours, Resolution.HOUR)