book
Checkout our new book! Hands on AI Trading with Python, QuantConnect, and AWS Learn More arrow

Custom Securities

Key Concepts

Introduction

To receive your custom data in the on_data method instead of in a bulk download, create a custom type and then create a data subscription. The custom data type tells LEAN where to get your data and how to read it. All custom data types must extend from PythonData and override the reader and get_source methods.

Unlike custom data, native QC data gets the full benefit of security modeling like splits and dividends. Custom data is ignorant of everything about the actual market. Things like market hours, delistings, and mergers don't work with custom data.

Create Subscriptions

After you define the custom data class, in the initialize method of your algorithm, call the add_data method. This method gives LEAN the type factory to create the objects, the name of the data, and the resolution at which to poll the data source for updates.

Select Language:
class MyAlgorithm(QCAlgorithm): 
    def initialize(self) -> None:
        self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).symbol

The resolution argument should match the resolution of your custom dataset. The lowest reasonable resolution is every minute. Anything more frequent than every minute is very slow to execute. The frequency that LEAN checks the data source depends on the resolution argument. The following table shows the polling frequency of each resolution:

ResolutionUpdate Frequency
DailyEvery 30 minutes
HourEvery 30 minutes
MinuteEvery minute
SecondEvery second
TickConstantly checks for new data

There are several other signatures for the add_data method.

self.add_data(type, ticker, resolution)
self.add_data(type, ticker, resolution, time_zone, fill_forward, leverage)
self.add_data(type, ticker, properties, exchange_hours, resolution, fill_forward, leverage)

Receive Custom Data

As your data reader reads your custom data file, LEAN adds the data points in the Slice it passes to your algorithm's on_data method. To collect the custom data, use the symbol or name of your custom data subscription. You can access the value and custom properties of your custom data class from the Slice. To access the custom properties, pass the property name to the get_property method.

Select Language:
class MyAlgorithm(QCAlgorithm):
    def on_data(self, slice: Slice) -> None:
        if slice.contains_key(self._symbol):
            custom_data = slice[self._symbol]
            value = custom_data.value
            property1 = custom_data.property1

Debugging

In backtests, you can use the IDE debugging capabilities to inspect the variables to understand the internal state of the custom data implementation. For more information about backtest debugging, see the Debugging documentation for Cloud Platform, Local Platform, or the CLI.

In live trading, consider adding logging statements to help with debugging since the debugger isn't available. To log information from inside your custom data class, set a class member that references the algorithm.

Select Language:
class MyCustomDataTypeAlgorithm(QCAlgorithm):
    def initialize(self):
        MyCustomDataType.ALGORITHM = self
        self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).Symbol

class MyCustomDataType(PythonData):
    def reader(self, config: SubscriptionDataConfig, line: str, date: datetime, is_live_mode: bool) -> BaseData:
        if not line[0].isdigit():
            # Display the line with the header
            MyCustomDataType.ALGORITHM.Debug(f"HEADER: {line}");
            return None
        data = line.split(',')
        custom = MyCustomDataType()
        custom.symbol = config.symbol
        custom.end_time = datetime.strptime(data[0], '%Y%m%d') + timedelta(1)
        custom.value = float(data[1])
        return custom

Set the Benchmark

To set your custom data source as the benchmark, in the initialize method, call the set_benchmark method with the Symbol of your custom data subscription.

Select Language:
self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).symbol
self.set_benchmark(self._symbol)

Avoid Look-Ahead Bias

Look-ahead bias occurs when you make decisions with information that wouldn't be available until some time in the future. In backtesting, look-ahead bias occurs when you receive data earlier than it would actually be available in reality. If look-ahead bias seeps into your algorithm, it will perform differently between live trading and backtesting.

To avoid look-ahead bias, set the timestamp of data points to the time when the data would actually be available. A lot of external sources apply timestamps to data differently than we do. For instance, on some platforms, daily bars are displayed with the date that they opened. We display daily bars with the date they closed. If you set the end_time to the start of the bar, you'll receive the bar earlier in backtests than you would in live trading.

Time Modeling

Data types classes in LEAN inherit from the BaseData class that defines the time and end_time properties. These properties represent the time period of which the data was built. If the data type occurs in a singular point in time, they have no period, so time and end_time are the same. Regardless of the period, LEAN uses the time when the data sample ends, end_time, to add the sample to a Slice.

Historical Data

To get historical data points for a custom dataset, call the history method with the dataset Symbol. For more information about historical data, see Data Points.

Market Hours

The default market hours for custom securities is to be always open. To set the market hours, pass an exchange_hours argument to the add_data method.

Select Language:
symbol = Symbol.create('SPY', SecurityType.EQUITY, Market.USA)
symbol_properties = self.symbol_properties_database.get_symbol_properties(symbol.id.market, symbol, SecurityType.EQUITY, 'USD')
security_exchange_hours = self.market_hours_database.get_exchange_hours(symbol.id.market, symbol, SecurityType.EQUITY)
security = self.add_data(CustomType, contract, symbol_properties, security_exchange_hours, Resolution.HOUR)

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: