Custom Securities

Key Concepts

Introduction

To receive your custom data in the OnDataon_data method instead of in a bulk download, create a custom type and then create a data subscription. The custom data type tells LEAN where to get your data and how to read it. All custom data types must extend from BaseDataPythonData and override the Readerreader and GetSourceget_source methods.

Unlike custom data, native QC data gets the full benefit of security modeling like splits and dividends. Custom data is ignorant of everything about the actual market. Things like market hours, delistings, and mergers don't work with custom data.

Create Subscriptions

After you define the custom data class, in the Initializeinitialize method of your algorithm, call the AddDataadd_data method. This method gives LEAN the type factory to create the objects, the name of the data, and the resolution at which to poll the data source for updates.

public class MyAlgorithm : QCAlgorithm
{
    private Symbol _symbol;
    public override void Initialize()
    {
        _symbol = AddData<MyCustomDataType>("<name>", Resolution.Daily).Symbol;
    }
}
class MyAlgorithm(QCAlgorithm): 
    def initialize(self) -> None:
        self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).symbol

The resolution argument should match the resolution of your custom dataset. The lowest reasonable resolution is every minute. Anything more frequent than every minute is very slow to execute. The frequency that LEAN checks the data source depends on the resolution argument. The following table shows the polling frequency of each resolution:

ResolutionUpdate Frequency
DailyEvery 30 minutes
HourEvery 30 minutes
MinuteEvery minute
SecondEvery second
TickConstantly checks for new data

There are several other signatures for the AddDataadd_data method.

self.add_data(type, ticker, resolution)
self.add_data(type, ticker, resolution, time_zone, fill_forward, leverage)
self.add_data(type, ticker, properties, exchange_hours, resolution, fill_forward, leverage)
AddData<T>(ticker, resolution);
AddData<T>(ticker, resolution, fillForward, leverage);
AddData<T>(ticker, resolution, timeZone, fillForward, leverage);
AddData<T>(ticker, properties, exchangeHours, resolution, fillForward, leverage);

Receive Custom Data

As your data reader reads your custom data file, LEAN adds the data points in the Slice it passes to your algorithm's OnDataon_data method. To collect the custom data, use the Symbolsymbol or name of your custom data subscription. You can access the Valuevalue and custom properties of your custom data class from the Slice. To access the custom properties, use the custom attributepass the property name to the GetPropertyget_property method.

public class MyAlgorithm : QCAlgorithm
{
    public override void OnData(Slice slice)
    {
        if (slice.ContainsKey(_symbol))
        {
            var customData = slice[_symbol];
            var value = customData.Value;
            var property1 = customData.Property1;
        }
    }

    // You can also get the data directly with OnData(<dataClass>) method
    public void OnData(MyCustomDataType slice)
    {
        var value = slice.Value;
        var property1 = slice.Property1;
    }
}
class MyAlgorithm(QCAlgorithm):
    def on_data(self, slice: Slice) -> None:
        if slice.contains_key(self._symbol):
            custom_data = slice[self._symbol]
            value = custom_data.value
            property1 = custom_data.property1

Debugging

In backtests, you can use the IDE debugging capabilities to inspect the variables to understand the internal state of the custom data implementation. For more information about backtest debugging, see the Debugging documentation for Cloud Platform, Local Platform, or the CLI.

In live trading, consider adding logging statements to help with debugging since the debugger isn't available. To log information from inside your custom data class, set a class member that references the algorithm. 

public class MyCustomDataTypeAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        MyCustomDataType.ALGORITHM = this;
        var symbol = AddData<MyCustomDataType>("<name>", Resolution.Daily).Symbol;
    }
}

public class MyCustomDataType : BaseData
{
    public static QCAlgorithm ALGORITHM;

    public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
    {
        if (!char.IsDigit(line.Trim()[0]))
        {
            // Display the line with the header
            ALGORITHM.Debug($"HEADER: {line}");
            return null;
        }
        var data = line.Split(',');
        return new MyCustomDataType()
        {
            Time = DateTime.ParseExact(data[0], "yyyyMMdd", CultureInfo.InvariantCulture),
            EndTime = Time.AddDays(1),
            Symbol = config.Symbol,
            Value = data[1].IfNotNullOrEmpty(
                s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture)),
        };
    }
}
class MyCustomDataTypeAlgorithm(QCAlgorithm):
    def initialize(self):
        MyCustomDataType.ALGORITHM = self
        self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).Symbol

class MyCustomDataType(PythonData):
    def reader(self, config: SubscriptionDataConfig, line: str, date: datetime, is_live_mode: bool) -> BaseData:
        if not line[0].isdigit():
            # Display the line with the header
            MyCustomDataType.ALGORITHM.Debug(f"HEADER: {line}");
            return None
        data = line.split(',')
        custom = MyCustomDataType()
        custom.symbol = config.symbol
        custom.end_time = datetime.strptime(data[0], '%Y%m%d') + timedelta(1)
        custom.value = float(data[1])
        return custom

Set the Benchmark

To set your custom data source as the benchmark, in the Initializeinitialize method, call the SetBenchmarkset_benchmark method with the Symbol of your custom data subscription.

var symbol = AddData<MyCustomDataType>("<name>", Resolution.Daily).Symbol;
SetBenchmark(symbol);
self._symbol = self.add_data(MyCustomDataType, "<name>", Resolution.DAILY).symbol
self.set_benchmark(self._symbol)

Avoid Look-Ahead Bias

Look-ahead bias occurs when you make decisions with information that wouldn't be available until some time in the future. In backtesting, look-ahead bias occurs when you receive data earlier than it would actually be available in reality. If look-ahead bias seeps into your algorithm, it will perform differently between live trading and backtesting.

To avoid look-ahead bias, set the timestamp of data points to the time when the data would actually be available. A lot of external sources apply timestamps to data differently than we do. For instance, on some platforms, daily bars are displayed with the date that they opened. We display daily bars with the date they closed. If you set the EndTimeend_time to the start of the bar, you'll receive the bar earlier in backtests than you would in live trading.

Time Modeling

Data types classes in LEAN inherit from the BaseData class that defines the Timetime and EndTimeend_time properties. These properties represent the time period of which the data was built. If the data type occurs in a singular point in time, they have no period, so Timetime and EndTimeend_time are the same. Regardless of the period, LEAN uses the time when the data sample ends, EndTimeend_time, to add the sample to a Slice.

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: