Securities

Requesting Data

Introduction

When you add a security to your algorithm, your algorithm creates a data subscription and receives data updates for that security or custom data source.

Add Assets

To create an Equity subscription, call the AddEquityadd_equity method and pass the ticker argument. The ticker argument represents the current ticker.

// Add the SPY ETF.
var equity = AddEquity("SPY");
# Add the SPY ETF.
equity = self.add_equity("SPY")

The AddEquityadd_equity method creates a subscription for a single Equity asset and adds it to your user-defined universe.

For more information about adding assets, see Asset Classes.

Add Universes

Universe selection is the process of selecting a basket of assets you may trade. When you add a universe to your algorithm, LEAN sends a large dataset into a filter function you define. LEAN automatically subscribes to these new assets and adds them to your algorithm. 

// Use the FundamentalFilterFunction to add the 100 most liquid stocks to the universe.
public class MyFundamentalUniverseAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        UniverseSettings.Asynchronous = true;
        AddUniverse(FundamentalFilterFunction);
    }

    // Get the top 100 stocks by dollar volume.
    private IEnumerable<Symbol> FundamentalFilterFunction(IEnumerable<Fundamental> fundamental)
    {
        return (from c in fundamental
            orderby c.DollarVolume descending
            select c.Symbol).Take(100);
    }
}
# Use the _fundamental_filter_function to add the 100 most liquid stocks to the universe.
class MyFundamentalUniverseAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        self.universe_settings.asynchronous = True
        self.add_universe(self._fundamental_filter_function)

    # Get the top 100 stocks by dollar volume.
    def _fundamental_filter_function(self, fundamental: List[Fundamental]) -> List[Symbol]:
        sorted_by_dollar_volume = sorted(fundamental, key=lambda x: x.dollar_volume, reverse=True) 
        return [c.symbol for c in sorted_by_dollar_volume[:100]]

For more information about universes, see Universes.

Resolutions

Resolution is the duration of time that's used to sample a data source. Market data is sampled and saved in disk with five resolutions. The Resolution enumeration has the following members:

  • Resolution.TickResolution.TICK
  • Resolution.SecondResolution.SECOND
  • Resolution.MinuteResolution.MINUTE
  • Resolution.HourResolution.HOUR
  • Resolution.DailyResolution.DAILY

For more information about the resolutions of each asset class, follow these steps:

  1. Open the Asset Classes documentation.
  2. Click an asset class.
  3. Click Requesting Data.
  4. Scroll down to the Resolutions section.

You can subscribe to daily resolution for an asset for updating indicator and subscribe to another asset for denser resolution for trading using the indicator.

public override void Initialize()
{
    AddEquity("VIX", Resolution.Daily);
    AddEquity("SPY", Resolution.Minute);
}
def initialize(self):
    self.add_equity("VIX", Resolution.DAILY)
    self.add_equity("SPY", Resolution.MINUTE)

To get data with multiple or customizable resolutions for an asset, use consolidators.

Fill Forward

Fill forward means if there is no data point for the current slice, LEAN uses the previous data point. Fill forward is the default data setting. If you disable fill forward, you may get stale fills or you may see trade volume as zero.

For more information about how to enable and disable fill forward for each asset class, follow these steps:

  1. Open the Asset Classes documentation.
  2. Click an asset class.
  3. Click Requesting Data.
  4. Scroll down to the Fill Forward section.

Example: Disable Fill Forward Data

For some illiquid securities or alternative data, using forward filled data might not be an accurate estimation nor usable information. To avoid this, disable fill forward.

public override void Initialize()
{
    AddData<BrainSentiment7Days>(symbol, fillForward: false);
}
def initialize(self):
    self.add_data(BrainSentiment7Days, symbol, fill_forward=False)

Security Changed Events

When you add and remove assets, LEAN notifies your algorithm through the OnSecuritiesChangedon_securities_changed event handler. The event handler receives a SecurityChanges object, which contains references to the added and removed securities. To access the added securities, iterate the changes.AddedSecuritieschanges.added_securities property. To get the removed securities, iterate the changes.RemovedSecuritieschanges.removed_securities property.

// Use OnSecuritiesChanged to know when an asset is added or removed.
public override void OnSecuritiesChanged(SecurityChanges changes)
{
    // Iterate through the added assets.
    foreach (var security in changes.AddedSecurities)
    {
        Debug($"{Time}: Added {security.Symbol}");
    }

    // Iterate through the removed assets.
    foreach (var security in changes.RemovedSecurities)
    {
        Debug($"{Time}: Removed {security.Symbol}");
        
        if (security.Invested)
        {
            Liquidate(security.Symbol, "Removed from Universe");
        }
    }
}
# Use on_securities_changed to know when an asset is added or removed.
def on_securities_changed(self, changes: SecurityChanges) -> None:
    # Iterate through the added assets.
    for security in changes.added_securities:
        self.debug(f"{self.time}: Added {security.symbol}")

    # Iterate through the removed assets.
    for security in changes.removed_securities:
        self.debug(f"{self.time}: Removed {security.symbol}")
        
        if security.invested:
            self.liquidate(security.symbol, "Removed from Universe")

The preceding example liquidates securities that leave the universe. You can use this event to create indicators when a new security enters the universe.

Remove Subscriptions

To remove a security subscription, call the RemoveSecurityremove_security method.

// Remove the security subscription.
RemoveSecurity(_symbol);
# Remove the security subscription.
self.remove_security(self._symbol)

The RemoveSecurityremove_security method cancels your open orders for the security and liquidates your holdings.

Examples

The following examples demonstrate some common practices on the subject of requesting data.

Example 1: Add Linked Alternative Data for Universe Constituents

Many traders select universe constituents and then allocate positions with two independent sets of logic. Position sizing usually requires additional information, like alternative data factors. To manage alternative data subscriptions for all the universe constituents, subscribe and unsubscribe from the data in the OnSecuritiesChangedon_securities_changed method. The following example demonstrates adds the KavoutCompositeFactorBundle when assets enter the universe. 

public class RequestSecuritiesDataAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        // Add a universe of the 10 most liquid US Equities.
        AddUniverse(Universe.DollarVolume.Top(10));
    }

    public override void OnData(Slice slice)
    {
        // Invest by the updated Kavout Factor data.
        var factorData = slice.Get<KavoutCompositeFactorBundle>();
        if (factorData.Count > 0)
        {
            // Obtain the top 10 factor sum score symbols.
            // Invest by equal weighting to dissipate capital risks.
            var topFactorTargets = factorData.OrderByDescending(x => 
                    x.Value.Growth + x.Value.ValueFactor + x.Value.Quality + x.Value.Momentum + x.Value.LowVolatility
                )
                .Take(10)
                .Select(x => new PortfolioTarget(x.Key.Underlying, 0.1m))
                .ToList();
            SetHoldings(topFactorTargets);
        }
    }

    public override void OnSecuritiesChanged(SecurityChanges changes)
    {
        // Iterate securities that entered the universe.
        foreach (dynamic security in changes.AddedSecurities)
        {
            // Subscribe to the factor data for the new security.
            // Save the alternative data symbol in the security object.
            security.AltDataSymbol = AddData<KavoutCompositeFactorBundle>(security.Symbol, Resolution.Daily).Symbol;
        }
        
        // Iterate securities that left the universe.
        foreach (dynamic security in changes.RemovedSecurities)
        {
            // Unsubscribe from the altnerative data updates.
            RemoveSecurity(security.AltDataSymbol);
        }
    }
}
class RequestSecuritiesDataAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        # Add a universe of the 10 most liquid US Equities.
        self.add_universe(self.universe.dollar_volume.top(10))

    def on_data(self, slice: Slice) -> None:
        # Invest by the updated Kavout Factor data.
        factor_data = slice.get(KavoutCompositeFactorBundle)
        if factor_data:
            sorted_by_factor_score = sorted(factor_data.items(),
                key=lambda x: x[1].growth + x[1].value_factor + x[1].quality + x[1].momentum + x[1].low_volatility,
                reverse=True)[:10]
            top_factor_targets = [PortfolioTarget(x[0].underlying, 0.1) for x in sorted_by_factor_score]
            self.set_holdings(top_factor_targets)

    def on_securities_changed(self, changes: SecurityChanges) -> None:
        # Iterate securities that entered the universe.
        for security in changes.added_securities:
            # Subscribe to the factor data for the new security.
            # Save the alternative data symbol in the security object.
            security.alt_data_symbol = self.add_data(KavoutCompositeFactorBundle, security.symbol, Resolution.DAILY).symbol
        
        # Iterate securities that left the universe.
        for security in changes.removed_securities:
            # Unsubscribe from the altnerative data updates.
            self.remove_security(security.alt_data_symbol)

Example 2: Add Linked Custom Data

QuantConnect Cloud has a lot of data, but not everything. To import some external data into your algorithm, define a custom data source. The following algorithm loads Bitstamp's BTCUSD trading activity from a CSV file and plots it: 

// Define the algorithm.
public class LinkedCustomDataExampleAlgorithm : QCAlgorithm
{
    public override void Initialize()
    {
        SetStartDate(2011, 9, 13);
        SetEndDate(2021, 6, 20);
        // Add Bitcoin and the custom dataset.
        dynamic btc = AddCrypto("BTCUSD");
        btc.dataSymbol = AddData<Bitstamp>(btc.Symbol.Value, Resolution.Daily).Symbol;
        // Plot the price from the custom dataset.
        PlotIndicator("BTC Price", Identity(btc.dataSymbol));
    }
}

// Define the custom data source.
public class Bitstamp : BaseData
{
    // Declare the properties of the custom dataset.
    public int Timestamp = 0;
    public decimal Open = 0;
    public decimal High = 0;
    public decimal Low = 0;
    public decimal Close = 0;
    public decimal Bid = 0;
    public decimal Ask = 0;
    public decimal WeightedPrice = 0;
    public decimal VolumeBTC = 0;
    public decimal VolumeUSD = 0;

    // Define the GetSource method to load the data from its location.
    public override SubscriptionDataSource GetSource(SubscriptionDataConfig config, DateTime date, bool isLiveMode)
    {
        var source = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv";
        // RemoteFile in the next line means the data file in online storage.
        return new SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile);
    }

    // Define the Reader method to parse the CSV rows.
    public override BaseData Reader(SubscriptionDataConfig config, string line, DateTime date, bool isLiveMode)
    {
        // Ignore empty rows.
        if (string.IsNullOrWhiteSpace(line.Trim()))
        {
            return null;
        }
        
        // Instantiate a new data object.
        var coin = new Bitstamp() {Symbol = config.Symbol};

        // Ignore non-numeric lines.
        if (!char.IsDigit(line[0]))
        {
            return null;
        }

        // Split the CSV data by its commas.
        var data = line.Split(',');
        coin.Value = data[4].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        // Ignore invalid data.
        if (coin.Value == 0)
        {
            return null;
        }

        // Parse the row.
        coin.Time = DateTime.Parse(data[0], CultureInfo.InvariantCulture);
        coin.EndTime = coin.Time.AddDays(1);
        coin.Open = data[1].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.High = data[2].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Low = data[3].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.VolumeBTC = data[5].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.VolumeUSD = data[6].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.WeightedPrice = data[7].IfNotNullOrEmpty(s => decimal.Parse(s, NumberStyles.Any, CultureInfo.InvariantCulture));
        coin.Close = coin.Value;
        return coin;
    }
}
# Define the algorithm.
class LinkedCustomDataExampleAlgorithm(QCAlgorithm):

    def initialize(self):
        self.set_start_date(2011, 9, 13)
        self.set_end_date(2021, 6, 20)
        # Add Bitcoin and the custom dataset.
        btc = self.add_crypto('BTCUSD')
        btc.data_symbol = self.add_data(Bitstamp, btc.symbol.value, Resolution.DAILY).symbol
        # Plot the price from the custom dataset.
        self.plot_indicator('BTC Price', self.identity(btc.data_symbol))

# Define the custom data source.
class Bitstamp(PythonData):
    # Define the get_source method to load the data from its location.
    def get_source(self, config, date, is_live_mode):
        source = "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/bitstampusd.csv"
        # REMOTE_FILE in the next line means the data file in online storage.
        return SubscriptionDataSource(source, SubscriptionTransportMedium.REMOTE_FILE)
    
    # Define the reader method to parse the CSV rows.
    def reader(self, config, line, date, is_live_mode):
        # Ignore empty rows.
        if not line.strip():
            return None
        
        # Instantiate a new data object.
        coin = Bitstamp()
        coin.symbol = config.symbol

        # Ignore non-numeric lines.
        if not line[0].isdigit():
            return None

        # Split the CSV row by its commas.
        data = line.split(',')

        # Ignore invalid data.
        coin.value = float(data[4])
        if coin.value == 0:
            return None

        # Parse the row.
        coin.time = datetime.strptime(data[0], "%Y-%m-%d")
        coin.end_time = coin.time + timedelta(1)
        coin["Open"] = float(data[1])
        coin["High"] = float(data[2])
        coin["Low"] = float(data[3])
        coin["Close"] = coin.value
        coin["VolumeBTC"] = float(data[5])
        coin["VolumeUSD"] = float(data[6])
        coin["WeightedPrice"] = float(data[7])
        return coin

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: