In This Page

Introduction

Get All Stored Data

Create Sample Data

Save Data

Read Data

Delete Data

Cache Data

Get File Path

Storage Quotas

Example for DataFrames

Example for Plotting

Example of Custom Data

Preserve Insights Between Deployments

Live Trading Considerations

Writing Algorithms

Object Store

Introduction

The Object Store is a file system that you can use in your algorithms to save, read, and delete data. The Object Store is organization-specific, so you can save or read data from the same Object Store in all of your organization's projects. The Object Store works like a key-value storage system where you can store regular strings, JSON encoded strings, XML encoded strings, and bytes. You can access the data you store in the Object Store from backtests, the Research Environment, and live algorithms.

Get All Stored Data

To get all of the keys and values in the Object Store, iterate through the ObjectStoreobject_store property.

Select Language:

foreach (var kvp in ObjectStore)
{
    var key = kvp.Key;
    var value = kvp.Value;
}

for kvp in self.object_store:
    key = kvp.key
    value = kvp.value

To iterate through just the keys in the Object Store, iterate through the Keyskeys property.

Select Language:

foreach (var key in ObjectStore.Keys)
{
    continue;
}

for key in self.object_store.keys:
    continue

Create Sample Data

You need some data to store data in the Object Store.

Follow these steps to create some sample data:

Create a dictionary.

var dictSample = new Dictionary<string, int> { {"One", 1}, {"Two", 2}, {"Three", 3} };

Create a string.

Select Language:

var stringSample = "My string";

string_sample = "My string"

Create a Bytes object.

Select Language:

var bytesSample = Encoding.UTF8.GetBytes("My String");

bytes_sample = str.encode("My String")

Convert the dictionary to an XML-formatted object.

var xmlSample = new XElement("sample",
    dictSample.Select(kvp => new XElement(kvp.Key, kvp.Value)));
Log(xmlSample.ToString());

Save Data

The Object Store saves objects under a key-value system. If you save objects in backtests, you can access them from the Research Environment. To avoid slowing down your backtests, save data once in the OnEndOfAlgorithmon_end_of_algorithm event handler. In live trading, you can save data more frequently like at the end of a Traintrain method or after universe selection.

If you run algorithms in QuantConnect Cloud, you need storage create permissions to save data in the Object Store.

If you don't have data to store, create some sample data.

You can save Bytes and string objects in the Object Store. To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.

Strings

To save a string object, call the Savesave or SaveStringsave_string method.

Select Language:

var saveSuccessful = ObjectStore.Save($"{ProjectId}/stringKey", stringSample);

save_successful = self.object_store.save(f"{self.project_id}/string_key", string_sample)

To save a JSON object, call the SaveJson<T> method. This method helps to serialize the data into JSON format.

var saveSuccessful = ObjectStore.SaveJson<Dictionary<string, int>>($"{ProjectId}/jsonKey", dictSample);

To save an XML-formatted object, call the SaveXml<T> method.

var saveSuccessful = ObjectStore.SaveXml<XElement>($"{ProjectId}/xmlKey", xmlSample);

Bytes

To save a Bytes object (for example, zipped data), call the SaveBytessave_bytes method.

Select Language:

var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey", bytesSample)

var zippedDataSample = Compression.ZipBytes(Encoding.UTF8.GetBytes(stringSample), "data");
var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey.zip", zippedDataSample);

save_successful = self.object_store.save_bytes(f"{self.project_id}/bytes_key", bytes_sample)

zipped_data_sample = Compression.zip_bytes(bytes(string_sample, "utf-8"), "data")
zip_save_successful = self.object_store.save_bytes(f"{self.project_id}/bytesKey.zip", zipped_data_sample)

Read Data

Read data from the Object Store to import algorithm variables between deployments, import data from the Research Environment, or load trained machine learning models. To read data from the Object Store, you need to provide the key you used to store the object.

You can load Bytes and string objects from the Object Store.

Before you read data from the Object Store, check if the key exists.

Select Language:

if (ObjectStore.ContainsKey(key))
{
    // Read data
}

if self.object_store.contains_key(key):
    # Read data

Strings

To read a string object, call the Readread or ReadStringread_string method.

Select Language:

var stringData = ObjectStore.Read($"{ProjectId}/stringKey");

string_data = self.object_store.read(f"{self.project_id}/string_key")

To read a JSON object, call the ReadJson<T> method.

var jsonData = ObjectStore.ReadJson<Dictionary<string, int>>($"{ProjectId}/jsonKey");

To read an XML-formatted object, call the ReadXml<T> method.

var xmlData = ObjectStore.ReadXml<XElement>($"{ProjectId}/xmlKey");

var dict = xmlData.Elements().ToDictionary(x => x.Name.LocalName, x => int.Parse(x.Value));

Bytes

To read a Bytes object, call the ReadBytesread_bytes method.

Select Language:

var bytesData = ObjectStore.ReadBytes($"{ProjectId}/bytesKey");

byte_data = self.object_store.read_bytes(f"{self.project_id}/bytes_key")

Delete Data

Delete objects in the Object Store to remove objects that you no longer need. If you run algorithms in QuantConnect Cloud, you need storage delete permissions to delete data from the Object Store.

To delete objects from the Object Store, call the Deletedelete method. Before you delete data, check if the key exists. If you try to delete an object with a key that doesn't exist in the Object Store, the method raises an exception.

Select Language:

if (ObjectStore.ContainsKey(key))
{
    ObjectStore.Delete(key);
}

if self.object_store.contains_key(key):
    self.object_store.delete(key)

To delete all of the content in the Object Store, iterate through all the stored data.

Select Language:

foreach (var kvp in ObjectStore)
{
    ObjectStore.Delete(kvp.Key);
}

for kvp in self.object_store:
    self.object_store.delete(kvp.key)

Cache Data

When you write to or read from the Object Store, the algorithm caches the data. The cache speeds up the algorithm execution because if you try to read the Object Store data again with the same key, it returns the cached data instead of downloading the data again. The cache speeds up execution, but it can cause problems if you are trying to share data between two nodes under the same Object Store key. For example, consider the following scenario:

You open project A and save data under the key 123.
You open project B and save new data under the same key 123.
In project A, you read the Object Store data under the key 123, expecting the data from project B, but you get the original data you saved in step #1 instead.

You get the data from step 1 instead of step 2 because the cache contains the data from step 1.

To clear the cache, call the Clear method.

Select Language:

ObjectStore.Clear();

self.object_store.clear()

Get File Path

To get the file path for a specific key in the Object Store, call the GetFilePathget_file_path method. If the key you pass to the method doesn't already exist in the Object Store, it's added to the Object Store.

Select Language:

var filePath = ObjectStore.GetFilePath(key);

file_path = self.object_store.get_file_path(key)

Storage Quotas

If you run algorithms locally, you can store as much data as your hardware will allow. If you run algorithms in QuantConnect Cloud, you must stay within your storage quota. If you need more storage space, edit your storage plan.

Example for DataFrames

Follow these steps to create a DataFrame, save it into the Object Store, and load it from the Object Store:

Get some historical data.

Select Language:

var spy = AddEquity("SPY").Symbol;
var history = History(Securities.Keys, 360, Resolution.Daily);

spy = self.add_equity("SPY").symbol
df = self.history(self.securities.keys, 360, Resolution.DAILY)

Create a DataFrame.

using Microsoft.Data.Analysis; // 

var columns = new DataFrameColumn[] {
    new DateTimeDataFrameColumn("Time", history.Select(x => (DateTime)x[spy].EndTime)),
    new DecimalDataFrameColumn("SPY Open", history.Select(x => (decimal)x[spy].Open)),
    new DecimalDataFrameColumn("SPY High", history.Select(x => (decimal)x[spy].High)),
    new DecimalDataFrameColumn("SPY Low", history.Select(x => (decimal)x[spy].Low)),
    new DecimalDataFrameColumn("SPY Close", history.Select(x => (decimal)x[spy].Close))
};
var df = new DataFrame(columns);

Get the file path for a specific key in the Object Store.

Select Language:

var filePath = ObjectStore.GetFilePath("df_to_csv");

file_path = self.object_store.get_file_path("df_to_csv")

Call the SaveCsv to_csv method to save the DataFrame in the Object Store as a CSV file.

Select Language:

DataFrame.SaveCsv(df, filePath);    // File size: 26520 bytes

df.to_csv(file_path)   # File size: 32721 bytes

Call the LoadCsv read_csv method to load the CSV file from the Object Store.

Select Language:

var reread = DataFrame.LoadCsv(filePath);

reread = pd.read_csv(file_path)

pandas supports saving and loading DataFrame objects in the following additional formats:

XML

file_path = self.object_store.get_file_path("df_to_xml")
df.to_xml(file_path)   # File size: 87816 bytes
reread = pd.read_xml(file_path)

JSON

file_path = self.object_store.get_file_path("df_to_json")
df.to_json(file_path)   # File size: 125250 bytes
reread = pd.read_json(file_path)

Parquet

file_path = self.object_store.get_file_path("df_to_parquet")
df.to_parquet(file_path)   # File size: 23996 bytes
reread = pd.read_parquet(file_path)

Pickle

file_path = self.object_store.get_file_path("df_to_pickle")
df.to_pickle(file_path)   # File size: 19868 bytes
reread = pd.read_pickle(file_path)

Example for Plotting

You can use the Object Store to plot data from your backtests and live algorithm in the Research Environment. The following example demonstrates how to plot a Simple Moving Average indicator that's generated during a backtest.

Create a algorithm, add a data subscription, and add a simple moving average indicator.

Select Language:

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private SimpleMovingAverage _sma;
    private string _content;

    public override void Initialize()
    {
        AddEquity("SPY", Resolution.Minute);
        _sma = SMA("SPY", 22);
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def initialize(self):
        self.add_equity("SPY")
    
        self.content = ''
        self._sma = self.sma("SPY", 22)

The algorithm will save _contentself.content to the Object Store.

Save the indicator data as string in _contentself.content.

Select Language:

public override void OnData(Slice data)
{
    _content += $"{_sma.Current.EndTime},{_sma}\n";
}

def on_data(self, data: Slice):
    self.plot('SMA', 'Value', self.sma.current.value)
    self.content += f'{self.sma.current.end_time},{self.sma.current.value}\n'

In the OnEndOfAlgorithm method, save the indicator data to the Object Store.

Select Language:

public override void OnEndOfAlgorithm()
{
    ObjectStore.Save("sma_values_csharp", _content);
}

def on_end_of_algorithm(self):
    self.object_store.save('sma_values_python', self.content)

Open the Research Environment and create a QuantBook.

Select Language:

// Execute the following command in first
#load "../Initialize.csx"

// Create a QuantBook object
#load "../QuantConnect.csx"
using QuantConnect;
using QuantConnect.Research;

var qb = new QuantBook();

qb = QuantBook()

Read the indicator data from the Object Store.

Select Language:

var content = qb.ObjectStore.Read("sma_values_csharp");

content = qb.object_store.read("sma_values_python")

The key you provide must be the same key you used to save the object.

Convert the data to a pandas object and create a chart.

data = {}
for line in content.split('\n'):
    csv = line.split(',')
    if len(csv) > 1:
        data[csv[0]] = float(csv[1])

series = pd.Series(data, index=data.keys())
series.plot()

Import the Plotly.NET and Plotly.NET.LayoutObjects packages.

#r "../Plotly.NET.dll"
using Plotly.NET;
using Plotly.NET.LayoutObjects;

Create the Layout object and set the title, xaxis, and yaxis properties.

var layout = new Layout();
layout.SetValue("title", Title.init("SMA"));

var xAxis = new LinearAxis();
xAxis.SetValue("title", "Time");
layout.SetValue("xaxis", xAxis);

var yAxis = new LinearAxis();
yAxis.SetValue("title", "SMA");
layout.SetValue("yaxis", yAxis);

Convert the data to a list of DateTime objects for the chart x-axis and a list of decimal objects for the chart y-axis, then create a Chart2D.Chart.Line object with the data.

var index = new List<DateTimee>();
var values = new List<decimal>();

foreach (var line in content.Split('\n'))
{
    var csv = line.Split(',');
    if (csv.Length > 1)
    {
        index.Add(Parse.DateTime(csv[0]));
        values.Add(decimal.Parse(csv[1]));
    }
}

var chart = Chart2D.Chart.Linee<DateTime, decimal, stringe>(index, values);

Apply the layout to the Line object and create the HTML object.

chart.WithLayout(layout);
var result = HTML(GenericChart.toChartHTML(chart));

Select Language:

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private SimpleMovingAverage _sma;
    private string _content;

    public override void Initialize()
    {
        SetStartDate(2023, 1, 1);   //Set Start Date
        SetCash(100000);            //Set Strategy Cash
        
        AddEquity("SPY", Resolution.Minute);
        // Create SMA indicator for referencing.
        _sma = SMA("SPY", 22);
    }

    public override void OnData(Slice data)
    {
        // Cache the indicator data point to save it.
        _content += $"{_sma.Current.EndTime},{_sma}\n";
    }

    public override void OnEndOfAlgorithm()
    {
        // Save the indicator values to object store for logging.
        ObjectStore.Save("sma_values_csharp", _content);
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        self.set_start_date(2023, 1, 1)  # Set Start Date
        self.set_cash(100000)           # Set Strategy Cash
        self.add_equity("SPY", Resolution.MINUTE)
        
        self.content = ''
        # Create SMA indicator for referencing.
        self.sma = self.SMA("SPY", 22)
        
    def on_data(self, data: Slice) -> None:
        # Cache the indicator data point to save it.
        self.content += f'{self.sma.current.end_time},{self.sma.current.value}\n'

    def on_end_of_algorithm(self) -> None:
        # Save the indicator values to object store for logging.
        self.object_store.save('sma_values_python', self.content)

Example of Custom Data

Follow these steps to use the Object Store as a data source for custom data:

Create a custom data class that defines a storage key and implements the GetSourceget_source method.

Select Language:

public class Bitstamp : TradeBar
{
    public static string KEY = "bitstampusd.csv";
            
    public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
    {
        return new SubscriptionDataSource(KEY, SubscriptionTransportMedium.ObjectStore);
    }
}

class Bitstamp(PythonData):
    KEY = 'bitstampusd.csv'
    def get_source(self, config, date, isLiveMode):
        return SubscriptionDataSource(Bitstamp.KEY, SubscriptionTransportMedium.OBJECT_STORE)

Create an algorithm that downloads data from an external source and saves it to the Object Store.

Select Language:

public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        if (!ObjectStore.ContainsKey(Bitstamp.KEY))
        {
            var url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv";
            var content = Download(url);
            ObjectStore.Save(Bitstamp.KEY, content);
        }   
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self):
        if not self.object_store.contains_key(Bitstamp.KEY):
            url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv"
            content = self.download(url)
            self.object_store.save(Bitstamp.KEY, content)

Call the AddDataadd_data method to subscribe to the custom type.

Select Language:

public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
{
    private Symbol _customDataSymbol;
    public override void Initialize()
    {
        _customDataSymbol = AddData<Bitstamp>("BTC").Symbol;  
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self):
        self.custom_data_symbol = self.add_data(Bitstamp, "BTC").symbol

Implement the Reader method for the custom data class.

Select Language:

public class Bitstamp : TradeBar
{
    public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
    {
        //Example Line Format:
        //Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
        //2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
        if (!char.IsDigit(line.Trim()[0]))
        {
            return null;
        }

        var coin = new Bitstamp() {Symbol = config.Symbol};

        var data = line.Split(',');
        coin.Value = data[4].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        if (coin.Value == 0)
        {
            return null;
        }

        coin.Time = DateTime.Parse(data[0], CultureInfo.InvariantCulture);
        coin.EndTime = coin.Time.AddDays(1);
        coin.Open = data[1].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.High = data[2].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Low = data[3].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.VolumeBTC = data[5].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Volume = data[6].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.WeightedPrice = data[7].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Close = coin.Value;
        return coin;
    }
}

class Bitstamp(PythonData):
    def reader(self, config, line, date, isLiveMode):
        # Example Line Format:
        # Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
        # 2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
        if not line[0].isdigit():
            return None

        coin = Bitstamp()
        coin.symbol = config.symbol
        data = line.split(',')

        # If value is zero, return None
        coin.value = float(data[4])
        if coin.value == 0:
            return None

        coin.time = datetime.strptime(data[0], "%Y-%m-%d")
        coin.end_time = coin.time + timedelta(1)
        coin["Open"] = float(data[1])
        coin["High"] = float(data[2])
        coin["Low"] = float(data[3])
        coin["Close"] = coin.value
        coin["VolumeBTC"] = float(data[5])
        coin["VolumeUSD"] = float(data[6])
        coin["WeightedPrice"] = float(data[7])
        return coin

To confirm your algorithm is receiving the custom data, request some historical data, then log and plot the data from the Slice object.

Select Language:

public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        var history = History(_customDataSymbol, 200, Resolution.Daily);
        Debug($"We got {history.Count()} items from historical data request of {_customDataSymbol}.");
    }
    public void OnData(Bitstamp data)
    {
        Log($"{data.EndTime}: Close: {data.Close}");
        Plot(_customDataSymbol, "Price", data.Close);
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self):
        history = self.history(Bitstamp, self.custom_data_symbol, 200, Resolution.DAILY)
        self.debug(f"We got {len(history)} items from historical data request of {self.custom_data_symbol}.")
    def on_data(self, slice):
        data = slice.get(Bitstamp).get( self.custom_data_symbol)
        if not data:
            return

        self.log(f'{data.end_time}: Close: {data.close}')
        self.plot(self.custom_data_symbol, 'Price', data.close)

The following algorithm provides a full example of a sourcing custom data from the Object Store:

Select Language:

public class CustomDataBitstampAlgorithm : QCAlgorithm
{
    private Symbol _customDataSymbol;

    public override void Initialize()
    {
        SetStartDate(2012, 9, 13);
        SetEndDate(2021, 6, 20);

        // Load the custom data from ObjectStore.
        if (!ObjectStore.ContainsKey(Bitstamp.KEY))
        {
            var url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv";
            var content = Download(url);
            ObjectStore.Save(Bitstamp.KEY, content);
        }

        // Request custom data for plotting.
        // Define the symbol and "type" of our generic data.
        _customDataSymbol = AddData<Bitstamp>("BTC", Resolution.Daily).Symbol;

        // Example historical call.
        var history = History(_customDataSymbol, 200, Resolution.Daily);
        Debug($"We got {history.Count()} items from historical data request of {_customDataSymbol}.");
    }

    public void OnData(Bitstamp data)
    {
        // Plot on updated data.
        Log($"{data.EndTime}: Close: {data.Close}");
        Plot(_customDataSymbol, "Price", data.Close);
    }

    public class Bitstamp : TradeBar
    {
        public static string KEY = "bitstampusd.csv";
        
        public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
        {
            // Load data from object store.
            return new SubscriptionDataSource(KEY, SubscriptionTransportMedium.ObjectStore);
        }

        public decimal WeightedPrice = 0;
        public decimal VolumeBTC = 0;

        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
        {
            //Example Line Format:
            //Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
            //2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
            if (!char.IsDigit(line.Trim()[0]))
            {
                return null;
            }

            var coin = new Bitstamp() {Symbol = config.Symbol};

            var data = line.Split(',');
            coin.Value = data[4].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            if (coin.Value == 0)
            {
                return null;
            }

            coin.Time = DateTime.Parse(data[0], CultureInfo.InvariantCulture);
            coin.EndTime = coin.Time.AddDays(1);
            coin.Open = data[1].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.High = data[2].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.Low = data[3].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.VolumeBTC = data[5].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.Volume = data[6].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.WeightedPrice = data[7].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.Close = coin.Value;
            return coin;
        }
    }
}

class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        self.set_start_date(2012, 9, 13)
        self.set_end_date(2021, 6, 20)
        self.set_cash(100000)

        # Load the custom data from ObjectStore.
        if not self.object_store.contains_key(Bitstamp.KEY):
            url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv"
            content = self.download(url)
            self.object_store.save(Bitstamp.KEY, content)

        # Request custom data for plotting.
        # Define the symbol and "type" of our generic data.
        self.custom_data_symbol = self.add_data(Bitstamp, "BTC").symbol

        # Example historical call.
        history = self.history(Bitstamp, self.custom_data_symbol, 200, Resolution.DAILY)
        self.debug(f"We got {len(history)} items from historical data request of {self.custom_data_symbol}.")

    def on_data(self, slice: Slice) -> None:
        # Plot on updated data.
        data = slice.get(Bitstamp).get(self.custom_data_symbol)
        if not data:
            return

        self.log(f'{data.end_time}: Close: {data.close}')
        self.plot(self.custom_data_symbol, 'Price', data.close)


class Bitstamp(PythonData):
    KEY = 'bitstampusd.csv'
    def get_source(self, config, date, isLiveMode):
        # Load data from object store.
        return SubscriptionDataSource(Bitstamp.KEY, SubscriptionTransportMedium.OBJECT_STORE)

    def reader(self, config, line, date, isLiveMode):
        # Example Line Format:
        # Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
        # 2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
        if not line[0].isdigit():
            return None

        coin = Bitstamp()
        coin.symbol = config.symbol
        data = line.split(',')

        # If value is zero, return None
        coin.value = float(data[4])
        if coin.value == 0:
            return None

        coin.time = datetime.strptime(data[0], "%Y-%m-%d")
        coin.end_time = coin.time + timedelta(1)
        coin["Open"] = float(data[1])
        coin["High"] = float(data[2])
        coin["Low"] = float(data[3])
        coin["Close"] = coin.value
        coin["VolumeBTC"] = float(data[5])
        coin["VolumeUSD"] = float(data[6])
        coin["WeightedPrice"] = float(data[7])
        return coin

Preserve Insights Between Deployments

Follow these steps to use the Object Store to preserve the algorithm state across live deployments:

Create an algorithm that defines a storage key and adds insights to the Insight Manager.

Select Language:

public class ObjectStoreChartingAlgorithm : QCAlgorithm
{
    private string _insightKey;
    public override void Initialize()
    {
        _insightKey = $"{ProjectId}/insights";
        SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
        SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));    
    }
}

class ObjectStoreChartingAlgorithm(QCAlgorithm):
    def initialize(self):
        self.insight_key = f"{self.project_id}/insights"
        self.set_universe_selection(ManualUniverseSelectionModel([ Symbol.create("SPY", SecurityType.EQUITY, Market.USA) ]))
        self.set_alpha(ConstantAlphaModel(InsightType.PRICE, InsightDirection.UP, timedelta(5), 0.025, None))

At the top of the algorithm file, add the following imports:

from Newtonsoft.Json import JsonConvert
from System.Collections.Generic import List

Insight objects are a C# objects, so you need the preceding C# libraries to serialize and deserialize them.

In the OnEndOfAlgorithmon_end_of_algorithm event handler of the algorithm, get the Insight objects and save them in the Object Store as a JSON object.

Select Language:

public override void OnEndOfAlgorithm()
{
    var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
    ObjectStore.SaveJson(_insightKey, insights);
}

def on_end_of_algorithm(self):
    insights = self.insights.get_insights(lambda x: x.is_active(self.utc_time))
    content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
    self.object_store.save(self.insight_key, f'[{content}]')

At the bottom of the Initializeinitialize method, read the Insight objects from the Object Store and add them to the Insight Manager.

Select Language:

if (ObjectStore.ContainsKey(_insightKey))
{
    var insights = ObjectStore.ReadJson<List<Insight>>(_insightKey);
    Insights.AddRange(insights);
}

if self.object_store.contains_key(self.insight_key):
    insights = self.object_store.read_json[List[Insight]](self.insight_key)
    self.insights.add_range(insights)

The following algorithm provides a full example of preserving the Insight state between deployments:

Select Language:

public class ObjectStoreInsightsAlgorithm : QCAlgorithm
{
    private string _insightsKey = "insights";

    public override void Initialize()
    {
        UniverseSettings.Resolution = Resolution.Daily;

        SetStartDate(2023, 4, 1);
        SetEndDate(2023, 4, 11);
        SetCash(100000);

        _insightsKey = $"{ProjectId}/{_insightsKey}";

        // Read the file with the insights
        if (ObjectStore.ContainsKey(_insightsKey))
        {
            // Load cached JSON key value pair from object store, then add to algorithm insights.
            var insights = ObjectStore.ReadJson<List<Insight>>(_insightsKey);
            Log($"Read {insights.Count} insight(s) from the Object Store");
            Insights.AddRange(insights);

            // Delete the key to reuse it
            ObjectStore.Delete(_insightsKey);
        }

        SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
        SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));
        SetPortfolioConstruction(new EqualWeightingPortfolioConstructionModel(Resolution.Daily));
    }

    public override void OnEndOfAlgorithm()
    {
        // Get all active insights.
        var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
        // If we want to save all insights (expired and active), we can use.
        // var insights = Insights.GetInsights(x => true);
        Log($"Save {insights.Count} insight(s) to the Object Store.");
        ObjectStore.SaveJson(_insightsKey, insights);
    }
}

class ObjectStoreInsightsAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        self.universe_settings.resolution = Resolution.DAILY

        self.set_start_date(2023,4,1)
        self.set_end_date(2023,4,11)
        self.set_cash(100000)

        self.insights_key = f"{self.project_id}/insights"

        # Read the file with the insights
        if self.object_store.contains_key(self.insights_key):
            # Load cached JSON key value pair from object store, then add to algorithm insights.
            insights = self.object_store.read_json[List[Insight]](self.insights_key)
            self.log(f"Read {len(insights)} insight(s) from the Object Store")
            self.insights.add_range(insights)

            # Delete the key to reuse it
            self.object_store.delete(self.insights_key)

        self.set_universe_selection(ManualUniverseSelectionModel([ Symbol.create("SPY", SecurityType.EQUITY, Market.USA) ]))
        self.set_alpha(ConstantAlphaModel(InsightType.PRICE, InsightDirection.UP, timedelta(5), 0.025, None))
        self.set_portfolio_construction(EqualWeightingPortfolioConstructionModel(Resolution.DAILY))

    def on_end_of_algorithm(self) -> None:
        # Get all active insights.
        insights = self.insights.get_insights(lambda x: x.is_active(self.utc_time))
        # If we want to save all insights (expired and active), we can use
        # insights = self.insights.get_insights(lambda x: True)
        self.log(f"Save {len(insights)} insight(s) to the Object Store.")
        content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
        self.object_store.save(self.insights_key, f'[{content}]')

Live Trading Considerations

If you update the Object Store from outside of your live trading node and then try to read the new content from inside your live trading algorithm, you may not get the new content because the Object Store caches data to speed up execution. To ensure you get the latest value for a specific key in the Object Store, clear the cache and then read the data.

Select Language:

ObjectStore.Clear();
if (ObjectStore.ContainsKey(key))
{
    var value = ObjectStore.Read(key);
}

self.object_store.clear()
if self.object_store.contains_key(key):
    value = self.object_store.read(key)

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation:

SIGN IN

Browse

Cloud Platform