Importing Data

Bulk Downloads

Introduction

There are two techniques to import data into your algorithm. You can either manually import the entire file or stream the file line-by-line into your algorithm's OnDataon_data event. This page explores importing an entire file for manual use.

Instead of downloading the file from a remote file provider, you can upload the file to the Object Store (with the Algorithm Lab or with the CLI) for faster execution.

Recommended Use Cases

The batch import technique is outside of the LEAN's awareness or control, so it can't enforce good practices. However, the batch import technique is good for the loading the following datasets:

  • Loading data into the Object Store
  • Trained AI Models
  • Well-defined historical price datasets
  • Parameters and setting imports such as Symbol lists

Download Files

The Downloaddownload method downloads the content served from a local file or URL and then returns it as a string.

Basic Usage

var file = Download("<filePathOrURL>");

file = self.download("<filePathOrURL>")

# If your file is in CSV format, convert it to a DataFrame with the `read_csv` method.
from io import StringIO
import pandas as pd
df = pd.read_csv(StringIO(file))

# If your file is in JSON format, parse it with the `loads` method.
import json
data = json.loads(file)

# If your file is in XML format, parse it with the `fromstring` method.
import xml.etree.ElementTree as ET
root = ET.fromstring(file)

Download Method Arguments

The Downloaddownload method can accept header settings, a username, and a password for authentication.

ArgumentData TypeDescriptionDefault Value
addressstringstrA string containing the URI to download
headers IEnumerable<KeyValuePair<string, string>> Dict[str,str]Defines header values to add to the requestEnumerable.Empty<KeyValuePair<string, string>>()dict()
userNameuser_namestringstrThe user name associated with the credentialsnullNone
passwordstringstrThe password for the user name associated with the credentialsnullNone

Download Request Headers

var headers = new Dictionary { { "1", "1" } };
Download(address, headers);
Download(address, headers, userName, password);

header = { "1": "1" }
self.download(address, headers)
self.download(address, headers, user_name, password)

Save Files

When you download a remote file, save it into the Object Store so that you don't have to download the file again. If you need to import the file multiple times, it's faster to import it from the Object Store rather than repeatedly downloading the file from the remote file provider.

Transport Binary Data

Follow these steps to transport binary files:

  1. Add the following imports to your local program:
    2. import pickle
import base64
  2. Serialize your object.
    3. pickle_bytes = pickle.dumps(my_object)
base64_str = base64.b64encode(pickle_bytes).decode('ascii')
  3. Save the string representation of your object to one of the supported sources.
  4. Download the remote file into your project.
    5. base64_str = self.download("<fileURL>")
  5. Restore the object.
    6. base64_bytes = base64_str.encode('ascii')
model = base64.b64decode(base64_bytes)
restored_model = pickle.loads(model)

Examples

Demonstration Algorithms
DropboxUniverseSelectionAlgorithm.cs Python DropboxCoarseFineAlgorithm.py Python NLTKSentimentTradingAlgorithm.py Python

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: