Writing Algorithms

Object Store

Introduction

The Object Store is a file system that you can use in your algorithms to save, read, and delete data. The Object Store is organization-specific, so you can save or read data from the same Object Store in all of your organization's projects. The Object Store works like a key-value storage system where you can store regular strings, JSON encoded strings, XML encoded strings, and bytes. You can access the data you store in the Object Store from backtests, the Research Environment, and live algorithms.

Get All Stored Data

To get all of the keys and values in the Object Store, iterate through the ObjectStoreobject_store property.

foreach (var kvp in ObjectStore)
{
    var key = kvp.Key;
    var value = kvp.Value;
}
for kvp in self.object_store:
    key = kvp.key
    value = kvp.value

To iterate through just the keys in the Object Store, iterate through the Keyskeys property.

foreach (var key in ObjectStore.Keys)
{
    continue;
}
for key in self.object_store.keys:
    continue

Create Sample Data

You need some data to store data in the Object Store.

Follow these steps to create some sample data:

  1. Create a dictionary.
  2. var dictSample = new Dictionary<string, int> { {"One", 1}, {"Two", 2}, {"Three", 3} };
  3. Create a string.
  4. var stringSample = "My string";
    string_sample = "My string"
  5. Create a Bytes object.
  6. var bytesSample = Encoding.UTF8.GetBytes("My String");
    bytes_sample = str.encode("My String")
  7. Convert the dictionary to an XML-formatted object.
  8. var xmlSample = new XElement("sample",
        dictSample.Select(kvp => new XElement(kvp.Key, kvp.Value)));
    Log(xmlSample.ToString());
    Sample XML format data

Save Data

The Object Store saves objects under a key-value system. If you save objects in backtests, you can access them from the Research Environment. To avoid slowing down your backtests, save data once in the OnEndOfAlgorithmon_end_of_algorithm event handler. In live trading, you can save data more frequently like at the end of a Traintrain method or after universe selection.

If you run algorithms in QuantConnect Cloud, you need storage create permissions to save data in the Object Store.

If you don't have data to store, create some sample data.

You can save the following types of objects in the Object Store:

  • Bytes objects
  • string objects
  • JSON objects
  • XML-formatted objects

You can save Bytes and string objects in the Object Store. To store data, you need to provide a key. If you provide a key that is already in the Object Store, it will overwrite the data at that location. To avoid overwriting objects from other projects in your organization, prefix the key with your project ID. You can find the project ID in the URL of your browser when you open a project. For example, the ID of the project at quantconnect.com/project/12345 is 12345.

Strings

To save a string object, call the Savesave or SaveStringsave_string method.

var saveSuccessful = ObjectStore.Save($"{ProjectId}/stringKey", stringSample);
save_successful = self.object_store.save(f"{self.project_id}/string_key", string_sample)

JSON

To save a JSON object, call the SaveJson<T> method. This method helps to serialize the data into JSON format.

var saveSuccessful = ObjectStore.SaveJson<Dictionary<string, int>>($"{ProjectId}/jsonKey", dictSample);

XML

To save an XML-formatted object, call the SaveXml<T> method.

var saveSuccessful = ObjectStore.SaveXml<XElement>($"{ProjectId}/xmlKey", xmlSample);

Bytes

To save a Bytes object (for example, zipped data), call the SaveBytessave_bytes method.

var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey", bytesSample)

var zippedDataSample = Compression.ZipBytes(Encoding.UTF8.GetBytes(stringSample), "data");
var saveSuccessful = ObjectStore.SaveBytes($"{ProjectId}/bytesKey.zip", zippedDataSample);
save_successful = self.object_store.save_bytes(f"{self.project_id}/bytes_key", bytes_sample)

zipped_data_sample = Compression.zip_bytes(bytes(string_sample, "utf-8"), "data")
zip_save_successful = self.object_store.save_bytes(f"{self.project_id}/bytesKey.zip", zipped_data_sample)

Read Data

Read data from the Object Store to import algorithm variables between deployments, import data from the Research Environment, or load trained machine learning models. To read data from the Object Store, you need to provide the key you used to store the object.

You can load the following types of objects from the Object Store:

  • Bytes objects
  • string objects
  • JSON objects
  • XML-formatted objects

You can load Bytes and string objects from the Object Store.

Before you read data from the Object Store, check if the key exists.

if (ObjectStore.ContainsKey(key))
{
    // Read data
}
if self.object_store.contains_key(key):
    # Read data

Strings

To read a string object, call the Readread or ReadStringread_string method.

var stringData = ObjectStore.Read($"{ProjectId}/stringKey");
string_data = self.object_store.read(f"{self.project_id}/string_key")

JSON

To read a JSON object, call the ReadJson<T> method.

var jsonData = ObjectStore.ReadJson<Dictionary<string, int>>($"{ProjectId}/jsonKey");

XML

To read an XML-formatted object, call the ReadXml<T> method.

var xmlData = ObjectStore.ReadXml<XElement>($"{ProjectId}/xmlKey");

If you created the XML object from a dictionary, reconstruct the dictionary.

var dict = xmlData.Elements().ToDictionary(x => x.Name.LocalName, x => int.Parse(x.Value));

Bytes

To read a Bytes object, call the ReadBytesread_bytes method.

var bytesData = ObjectStore.ReadBytes($"{ProjectId}/bytesKey");
byte_data = self.object_store.read_bytes(f"{self.project_id}/bytes_key")

Delete Data

Delete objects in the Object Store to remove objects that you no longer need. If you run algorithms in QuantConnect Cloud, you need storage delete permissions to delete data from the Object Store.

To delete objects from the Object Store, call the Deletedelete method. Before you delete data, check if the key exists. If you try to delete an object with a key that doesn't exist in the Object Store, the method raises an exception.

if (ObjectStore.ContainsKey(key))
{
    ObjectStore.Delete(key);
}
if self.object_store.contains_key(key):
    self.object_store.delete(key)

To delete all of the content in the Object Store, iterate through all the stored data.

foreach (var kvp in ObjectStore)
{
    ObjectStore.Delete(kvp.Key);
}
for kvp in self.object_store:
    self.object_store.delete(kvp.key)

Cache Data

When you write to or read from the Object Store, the algorithm caches the data. The cache speeds up the algorithm execution because if you try to read the Object Store data again with the same key, it returns the cached data instead of downloading the data again. The cache speeds up execution, but it can cause problems if you are trying to share data between two nodes under the same Object Store key. For example, consider the following scenario:

  1. You open project A and save data under the key 123.
  2. You open project B and save new data under the same key 123.
  3. In project A, you read the Object Store data under the key 123, expecting the data from project B, but you get the original data you saved in step #1 instead.
  4. You get the data from step 1 instead of step 2 because the cache contains the data from step 1.

To clear the cache, call the Clear method.

ObjectStore.Clear();
self.object_store.clear()

Get File Path

To get the file path for a specific key in the Object Store, call the GetFilePathget_file_path method. If the key you pass to the method doesn't already exist in the Object Store, it's added to the Object Store.

var filePath = ObjectStore.GetFilePath(key);
file_path = self.object_store.get_file_path(key)

Storage Quotas

If you run algorithms locally, you can store as much data as your hardware will allow. If you run algorithms in QuantConnect Cloud, you must stay within your storage quota. If you need more storage space, edit your storage plan.

Example for DataFrames

Follow these steps to create a DataFrame, save it into the Object Store, and load it from the Object Store:

  1. Get some historical data.
  2. var spy = AddEquity("SPY").Symbol;
    var history = History(Securities.Keys, 360, Resolution.Daily);
    spy = self.add_equity("SPY").symbol
    df = self.history(self.securities.keys, 360, Resolution.DAILY)
  3. Create a DataFrame.
  4. using Microsoft.Data.Analysis; // 
    
    var columns = new DataFrameColumn[] {
        new DateTimeDataFrameColumn("Time", history.Select(x => (DateTime)x[spy].EndTime)),
        new DecimalDataFrameColumn("SPY Open", history.Select(x => (decimal)x[spy].Open)),
        new DecimalDataFrameColumn("SPY High", history.Select(x => (decimal)x[spy].High)),
        new DecimalDataFrameColumn("SPY Low", history.Select(x => (decimal)x[spy].Low)),
        new DecimalDataFrameColumn("SPY Close", history.Select(x => (decimal)x[spy].Close))
    };
    var df = new DataFrame(columns);
  5. Get the file path for a specific key in the Object Store.
  6. var filePath = ObjectStore.GetFilePath("df_to_csv");
    file_path = self.object_store.get_file_path("df_to_csv")
  7. Call the SaveCsvto_csv method to save the DataFrame in the Object Store as a CSV file.
  8. DataFrame.SaveCsv(df, filePath);    // File size: 26520 bytes
    df.to_csv(file_path)   # File size: 32721 bytes
  9. Call the LoadCsvread_csv method to load the CSV file from the Object Store.
  10. var reread = DataFrame.LoadCsv(filePath);
    reread = pd.read_csv(file_path)

pandas supports saving and loading DataFrame objects in the following additional formats:

  • XML
  • file_path = self.object_store.get_file_path("df_to_xml")
    df.to_xml(file_path)   # File size: 87816 bytes
    reread = pd.read_xml(file_path)
  • JSON
  • file_path = self.object_store.get_file_path("df_to_json")
    df.to_json(file_path)   # File size: 125250 bytes
    reread = pd.read_json(file_path)
  • Parquet
  • file_path = self.object_store.get_file_path("df_to_parquet")
    df.to_parquet(file_path)   # File size: 23996 bytes
    reread = pd.read_parquet(file_path)
  • Pickle
  • file_path = self.object_store.get_file_path("df_to_pickle")
    df.to_pickle(file_path)   # File size: 19868 bytes
    reread = pd.read_pickle(file_path)

Example for Plotting

You can use the Object Store to plot data from your backtests and live algorithm in the Research Environment. The following example demonstrates how to plot a Simple Moving Average indicator that's generated during a backtest.

  1. Create a algorithm, add a data subscription, and add a simple moving average indicator.
  2. public class ObjectStoreChartingAlgorithm : QCAlgorithm
    {
        private SimpleMovingAverage _sma;
        private string _content;
    
        public override void Initialize()
        {
            AddEquity("SPY", Resolution.Minute);
            _sma = SMA("SPY", 22);
        }
    }
    class ObjectStoreChartingAlgorithm(QCAlgorithm):
        def initialize(self):
            self.add_equity("SPY")
        
            self.content = ''
            self._sma = self.sma("SPY", 22)

    The algorithm will save _contentself.content to the Object Store.

  3. Save the indicator data as string in _contentself.content.
  4. public override void OnData(Slice data)
    {
        _content += $"{_sma.Current.EndTime},{_sma}\n";
    }
    def on_data(self, data: Slice):
        self.plot('SMA', 'Value', self.sma.current.value)
        self.content += f'{self.sma.current.end_time},{self.sma.current.value}\n'
  5. In the OnEndOfAlgorithm method, save the indicator data to the Object Store.
  6. public override void OnEndOfAlgorithm()
    {
        ObjectStore.Save("sma_values_csharp", _content);
    }
    def on_end_of_algorithm(self):
        self.object_store.save('sma_values_python', self.content)
  7. Open the Research Environment and create a QuantBook.
  8. // Execute the following command in first
    #load "../Initialize.csx"
    
    // Create a QuantBook object
    #load "../QuantConnect.csx"
    using QuantConnect;
    using QuantConnect.Research;
    
    var qb = new QuantBook();
    qb = QuantBook()
  9. Read the indicator data from the Object Store.
  10. var content = qb.ObjectStore.Read("sma_values_csharp");
    content = qb.object_store.read("sma_values_python")

    The key you provide must be the same key you used to save the object.

  11. Convert the data to a pandas object and create a chart.
  12. data = {}
    for line in content.split('\n'):
        csv = line.split(',')
        if len(csv) > 1:
            data[csv[0]] = float(csv[1])
    
    series = pd.Series(data, index=data.keys())
    series.plot()
  13. Import the Plotly.NET and Plotly.NET.LayoutObjects packages.
  14. #r "../Plotly.NET.dll"
    using Plotly.NET;
    using Plotly.NET.LayoutObjects;
  15. Create the Layout object and set the title, xaxis, and yaxis properties.
  16. var layout = new Layout();
    layout.SetValue("title", Title.init("SMA"));
    
    var xAxis = new LinearAxis();
    xAxis.SetValue("title", "Time");
    layout.SetValue("xaxis", xAxis);
    
    var yAxis = new LinearAxis();
    yAxis.SetValue("title", "SMA");
    layout.SetValue("yaxis", yAxis);
  17. Convert the data to a list of DateTime objects for the chart x-axis and a list of decimal objects for the chart y-axis, then create a Chart2D.Chart.Line object with the data.
  18. var index = new List<DateTimee>();
    var values = new List<decimal>();
    
    foreach (var line in content.Split('\n'))
    {
        var csv = line.Split(',');
        if (csv.Length > 1)
        {
            index.Add(Parse.DateTime(csv[0]));
            values.Add(decimal.Parse(csv[1]));
        }
    }
    
    var chart = Chart2D.Chart.Linee<DateTime, decimal, stringe>(index, values);
  19. Apply the layout to the Line object and create the HTML object.
  20. chart.WithLayout(layout);
    var result = HTML(GenericChart.toChartHTML(chart));

Example of Custom Data

Follow these steps to use the Object Store as a data source for custom data:

  1. Create a custom data class that defines a storage key and implements the GetSourceget_source method.
  2. public class Bitstamp : TradeBar
    {
        public static string KEY = "bitstampusd.csv";
                
        public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
        {
            return new SubscriptionDataSource(KEY, SubscriptionTransportMedium.ObjectStore);
        }
    }    
    class Bitstamp(PythonData):
        KEY = 'bitstampusd.csv'
        def get_source(self, config, date, isLiveMode):
            return SubscriptionDataSource(Bitstamp.KEY, SubscriptionTransportMedium.OBJECT_STORE)
  3. Create an algorithm that downloads data from an external source and saves it to the Object Store.
  4. public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
    {
        public override void Initialize()
        {
            if (!ObjectStore.ContainsKey(Bitstamp.KEY))
            {
                var url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv";
                var content = Download(url);
                ObjectStore.Save(Bitstamp.KEY, content);
            }   
        }
    }
    class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
        def initialize(self):
            if not self.object_store.contains_key(Bitstamp.KEY):
                url = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv"
                content = self.download(url)
                self.object_store.save(Bitstamp.KEY, content)
  5. Call the AddDataadd_data method to subscribe to the custom type.
  6. public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
    {
        private Symbol _customDataSymbol;
        public override void Initialize()
        {
            _customDataSymbol = AddData<Bitstamp>("BTC").Symbol;  
        }
    }
    class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
        def initialize(self):
            self.custom_data_symbol = self.add_data(Bitstamp, "BTC").symbol
  7. Implement the Reader method for the custom data class.
  8. public class Bitstamp : TradeBar
    {
        public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
        {
            //Example Line Format:
            //Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
            //2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
            if (!char.IsDigit(line.Trim()[0]))
            {
                return null;
            }
    
            var coin = new Bitstamp() {Symbol = config.Symbol};
    
            var data = line.Split(',');
            coin.Value = data[4].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            if (coin.Value == 0)
            {
                return null;
            }
    
            coin.Time = DateTime.Parse(data[0], CultureInfo.InvariantCulture);
            coin.EndTime = coin.Time.AddDays(1);
            coin.Open = data[1].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.High = data[2].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.Low = data[3].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.VolumeBTC = data[5].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.Volume = data[6].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.WeightedPrice = data[7].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
            coin.Close = coin.Value;
            return coin;
        }
    }
    class Bitstamp(PythonData):
        def reader(self, config, line, date, isLiveMode):
            # Example Line Format:
            # Date      Open   High    Low     Close   Volume (BTC)    Volume (Currency)   Weighted Price
            # 2011-09-13 5.8    6.0     5.65    5.97    58.37138238,    346.0973893944      5.929230648356
            if not line[0].isdigit():
                return None
    
            coin = Bitstamp()
            coin.symbol = config.symbol
            data = line.split(',')
    
            # If value is zero, return None
            coin.value = float(data[4])
            if coin.value == 0:
                return None
    
            coin.time = datetime.strptime(data[0], "%Y-%m-%d")
            coin.end_time = coin.time + timedelta(1)
            coin["Open"] = float(data[1])
            coin["High"] = float(data[2])
            coin["Low"] = float(data[3])
            coin["Close"] = coin.value
            coin["VolumeBTC"] = float(data[5])
            coin["VolumeUSD"] = float(data[6])
            coin["WeightedPrice"] = float(data[7])
            return coin
  9. To confirm your algorithm is receiving the custom data, request some historical data, then log and plot the data from the Slice object.
  10. public class ObjectStoreCustomDataAlgorithm : QCAlgorithm
    {
        public override void Initialize()
        {
            var history = History(_customDataSymbol, 200, Resolution.Daily);
            Debug($"We got {history.Count()} items from historical data request of {_customDataSymbol}.");
        }
        public void OnData(Bitstamp data)
        {
            Log($"{data.EndTime}: Close: {data.Close}");
            Plot(_customDataSymbol, "Price", data.Close);
        }
    }
    class ObjectStoreCustomDataAlgorithm(QCAlgorithm):
        def initialize(self):
            history = self.history(Bitstamp, self.custom_data_symbol, 200, Resolution.DAILY)
            self.debug(f"We got {len(history)} items from historical data request of {self.custom_data_symbol}.")
        def on_data(self, slice):
            data = slice.get(Bitstamp).get( self.custom_data_symbol)
            if not data:
                return
    
            self.log(f'{data.end_time}: Close: {data.close}')
            self.plot(self.custom_data_symbol, 'Price', data.close)

The following algorithm provides a full example of a sourcing custom data from the Object Store:

Preserve Insights Between Deployments

Follow these steps to use the Object Store to preserve the algorithm state across live deployments:

  1. Create an algorithm that defines a storage key and adds insights to the Insight Manager.
  2. public class ObjectStoreChartingAlgorithm : QCAlgorithm
    {
        private string _insightKey;
        public override void Initialize()
        {
            _insightKey = $"{ProjectId}/insights";
            SetUniverseSelection(new ManualUniverseSelectionModel(QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA)));
            SetAlpha(new ConstantAlphaModel(InsightType.Price, InsightDirection.Up, TimeSpan.FromDays(5), 0.025, null));    
        }
    }
    class ObjectStoreChartingAlgorithm(QCAlgorithm):
        def initialize(self):
            self.insight_key = f"{self.project_id}/insights"
            self.set_universe_selection(ManualUniverseSelectionModel([ Symbol.create("SPY", SecurityType.EQUITY, Market.USA) ]))
            self.set_alpha(ConstantAlphaModel(InsightType.PRICE, InsightDirection.UP, timedelta(5), 0.025, None))
  3. At the top of the algorithm file, add the following imports:
  4. from Newtonsoft.Json import JsonConvert
    from System.Collections.Generic import List

    Insight objects are a C# objects, so you need the preceding C# libraries to serialize and deserialize them.

  5. In the OnEndOfAlgorithmon_end_of_algorithm event handler of the algorithm, get the Insight objects and save them in the Object Store as a JSON object.
  6. public override void OnEndOfAlgorithm()
    {
        var insights = Insights.GetInsights(x => x.IsActive(UtcTime));
        ObjectStore.SaveJson(_insightKey, insights);
    }
    def on_end_of_algorithm(self):
        insights = self.insights.get_insights(lambda x: x.is_active(self.utc_time))
        content = ','.join([JsonConvert.SerializeObject(x) for x in insights])
        self.object_store.save(self.insight_key, f'[{content}]')
  7. At the bottom of the Initializeinitialize method, read the Insight objects from the Object Store and add them to the Insight Manager.
  8. if (ObjectStore.ContainsKey(_insightKey))
    {
        var insights = ObjectStore.ReadJson<List<Insight>>(_insightKey);
        Insights.AddRange(insights);
    }
    if self.object_store.contains_key(self.insight_key):
        insights = self.object_store.read_json[List[Insight]](self.insight_key)
        self.insights.add_range(insights)

The following algorithm provides a full example of preserving the Insight state between deployments:

Live Trading Considerations

If you update the Object Store from outside of your live trading node and then try to read the new content from inside your live trading algorithm, you may not get the new content because the Object Store caches data to speed up execution. To ensure you get the latest value for a specific key in the Object Store, clear the cache and then read the data.

ObjectStore.Clear();
if (ObjectStore.ContainsKey(key))
{
    var value = ObjectStore.Read(key);
}
self.object_store.clear()
if self.object_store.contains_key(key):
    value = self.object_store.read(key)

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: