Custom Securities

JSON Format Example

Introduction

This page explains how to import custom data of a single security sourced in JSON format.

Data Format

The following code snippet shows an example JSON file

[
    {
      "Date": "1997-01-01",
      "Open": 905.2,
      "High": 941.4,
      "Low": 905.2,
      "Close": 939.55,
      "SharesTraded": 38948210,
      "Tur11er(Rs.Cr)": 978.21
    },
    {
      "Date": "1997-01-02",
      "Open": 941.95,
      "High": 944,
      "Low": 925.05,
      "Close": 927.05,
      "SharesTraded": 49118380,
      "Tur11er(Rs.Cr)": 1150.42
    },
    ...
    {
      "Date": "2014-07-25",
      "Open": 7828.2,
      "High": 7840.95,
      "Low": 7748.6,
      "Close": 7790.45,
      "SharesTraded": 153936037,
      "Tur11er(Rs.Cr)": 7827.61
    },
    {
      "Date": "2014-07-28",
      "Open": 7792.9,
      "High": 7799.9,
      "Low": 7722.65,
      "Close": 7748.7,
      "SharesTraded": 116534670,
      "Tur11er(Rs.Cr)": 6107.78
    }
]

The data in the file must be in chronological order.

Define Custom Types

To define a custom data type, inherit the BaseDataPythonData class and override the GetSource and Reader methods.

If you need to create multiple objects in your Readerreader method from a single line, follow these steps:

  1. In the GetSourceget_source method, pass FileFormat.UnfoldingCollectionFileFormat.UNFOLDING_COLLECTION as the third argument to the SubscriptionDataSource constructor.
  2. In the Readerreader method, order the objects by their timestamp and then return a BaseDataCollection(endTime, config.Symbol, objects)BaseDataCollection(end_time, config.symbol, objects) where objects is a list of your custom data objects.
using Newtonsoft.Json;
        
public class MyCustomDataType : BaseData
{
    [JsonProperty("Date")]
    public string Date;
    [JsonProperty("Open")]
    public decimal Open = 0;
    [JsonProperty("High")]
    public decimal High = 0;
    [JsonProperty("Low")]
    public decimal Low = 0;
    [JsonProperty("Close")]
    public decimal Close = 0;

    public override DateTime EndTime => Time.AddDays(1);

    public override SubscriptionDataSource GetSource(
        SubscriptionDataConfig config,
        DateTime date,
        bool isLiveMode)
    {
        if (!isLive)
        {
            return new SubscriptionDataSource("<customDataKey>", SubscriptionTransportMedium.ObjectStore, FileFormat.UnfoldingCollection);
        }
        return new SubscriptionDataSource("https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/unfolding-collection-example.json", SubscriptionTransportMedium.RemoteFile, FileFormat.UnfoldingCollection);
    }

    public override BaseData Reader(
        SubscriptionDataConfig config,
        string line,
        DateTime date,
        bool isLiveMode)
    {
        if (string.IsNullOrWhiteSpace(line.Trim()))
        {
            return null;
        }

        var objects = JsonConvert.DeserializeObject<List<MyCustomDataType>>(line).Select(index =>
        {
            index.Symbol = config.Symbol;
            index.Time = DateTime.Parse(x.Date, CultureInfo.InvariantCulture);
            index.Value = x.Close;
            return index;
        }).ToList();

        return new BaseDataCollection(objects.Last().EndTime, config.Symbol, objects);
    }
}
class MyCustomDataType(PythonData):
    def get_source(self,
         config: SubscriptionDataConfig,
         date: datetime,
         is_live: bool) -> SubscriptionDataSource:
        if not is_live:
            return SubscriptionDataSource("<custom_data_key>", SubscriptionTransportMedium.OBJECT_STORE, FileFormat.UNFOLDING_COLLECTION)
        return SubscriptionDataSource("https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/unfolding-collection-example.json", SubscriptionTransportMedium.REMOTE_FILE, FileFormat.UNFOLDING_COLLECTION)

    def reader(self,
         config: SubscriptionDataConfig,
         line: str,
         date: datetime,
         is_live: bool) -> BaseData:

        if not (line.strip()):
            return None

        objects = []
        data = json.loads(line)

        for datum in data:
            index = MyCustomDataType()
            index.symbol = config.symbol
            index.time = datetime.strptime(datum["Date"], "%Y-%m-%d")
            index.end_time = index.time + timedelta(1)
            index["Open"] = float(datum["Open"])
            index["High"] = float(datum["High"])
            index["Low"] = float(datum["Low"])
            index["Close"] = float(datum["Close"])
            index.value = index["Close"]
            objects.append(index)

        return BaseDataCollection(objects[-1].end_time, config.symbol, objects)

Create Subscriptions

To create a subscription for the custom type, in the Initializeinitialize method, call the AddDataadd_data method. The AddDataadd_data method returns a Security object, which contains a Symbol. Save a reference to the Symbolsymbol so you can use it in OnDataon_data to access the security data in the Slice.

public class MyAlgorithm : QCAlgorithm
{
    private Symbol _symbol;
    public override void Initialize()
    {
        _symbol = AddData<MyCustomDataType>("MyCustomDataType", Resolution.Daily).Symbol;
    }
}
class MyAlgorithm(QCAlgorithm): 
    def initialize(self) -> None:
        self._symbol = self.add_data(MyCustomDataType, "MyCustomDataType", Resolution.DAILY).symbol

The AddDataadd_data method creates a subscription for a single custom data asset and adds it to your user-defined universe.

Receive Custom Data

As your data reader reads your custom data file, LEAN adds the data points in the Slice it passes to your algorithm's OnDataon_data method. To collect the custom data, use the Symbolsymbol or name of your custom data subscription. You can access the Valuevalue and custom properties of your custom data class from the Slice. To access the custom properties, use the custom attributepass the property name to the GetPropertyget_property method.

public class MyAlgorithm : QCAlgorithm
{
    public override void OnData(Slice slice)
    {
        if (slice.ContainsKey(_symbol))
        {
            var customData = slice[_symbol];
            var close = customData.Close;
        }
    }

    // You can also get the data directly with OnData(<dataClass>) method
    public void OnData(MyCustomDataType customData)
    {
        var close = customData.Close;
    }
}
class MyAlgorithm(QCAlgorithm):
    def on_data(self, slice: Slice) -> None:
        if slice.contains_key(self._symbol):
            custom_data = slice[self._symbol]
            close = custom_data.close

If you add custom properties to your data object in the Readerreader method, LEAN adds them as members to the data object in the Slice. To ensure the property names you add in the Readerreader method follow the convention of member names, LEAN applies the following changes to the property names you provide in the Readerreader method:

  1. - and . characters are replaced with whitespace.
  2. The first letter is capitalized.
  3. Whitespace characters are removed.

For example, if you set a property name in the Readerreader method to ['some-property.name'], you can access it in the OnDataon_data method through the Somepropertyname member of your data object.

Demonstration Algorithms

CustomDataBitcoinAlgorithm.py Python CustomDataBitcoinAlgorithm.cs C#

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: