book
Checkout our new book! Hands on AI Trading with Python, QuantConnect, and AWS Learn More arrow

Universes

Key Concepts

Introduction

Universe selection is the process of selecting a basket of assets you may trade. Dynamic universe selection increase diversification and decrease selection bias in your algorithm.

How Universe Selection Works

When you add a universe to your algorithm, LEAN sends a large dataset into a filter function you define. Your filter function needs to return a list of Symbol objects. LEAN automatically subscribes to these new symbols and adds them to your algorithm. Your algorithm can do almost anything inside your filter functions, but the goal should be to narrow down the set of securities to the securities that are most applicable for your algorithm.

Selection Functions

The following example filter function selects the 100 most liquid US Equities:

Select Language:
# Use a fundamental filter function to select the 100 most liquid Equities.
class MyFundamentalUniverseAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        self.universe_settings.asynchronous = True
        self.add_universe(self._fundamental_filter_function)

    def _fundamental_filter_function(self, fundamental: List[Fundamental]) -> List[Symbol]:
        sorted_by_dollar_volume = sorted(fundamental, key=lambda x: x.dollar_volume, reverse=True)
        return [c.symbol for c in sorted_by_dollar_volume[:100]]

To learn how to define filter functions for other asset classes, custom data universes, or alternative data universes, see the following pages in this Universes chapter.

Security Changed Events

When your universe adds and removes assets, LEAN notifies your algorithm through the on_securities_changed event handler. The event handler receives a SecurityChanges object, which contains references to the added and removed securities. To access the added securities, check the changes.added_securities method property. To access the removed securities, check the changes.removed_securities method property.

Select Language:
# Use the on_securities_changed method to know when an asset is added or removed.
def on_securities_changed(self, changes: SecurityChanges) -> None:
    # Iterate through the added securities.
    for security in changes.added_securities:
        self.debug(f"{self.time}: Added {security.symbol}")

    # Iterate through the removed securities.
    for security in changes.removed_securities:
        self.debug(f"{self.time}: Removed {security.symbol}")
        
        if security.invested:
            self.liquidate(security.symbol, "Removed from Universe")

The preceding example liquidates securities that leave the universe. In this case, LEAN creates a market on open order and frees up buying power when the market opens.

Custom Security Properties

If you need to save data or create objects for each security in the universe, add custom members to the respective Security objects. This technique is useful if you want to track stop loss levels or add indicators for each asset in the universe.

Select Language:
# Add custom properties to the security by duck-typing.
def on_securities_changed(self, changes: SecurityChanges) -> None:
    for security in changes.added_securities:
        # Create an SMA indicator with 10 periods for the asset. 
        # Use duck typing to store it on the security object.
        security.indicator = self.sma(security.Symbol, 10)

        # Warm up the indicator with historical data.
        self.warm_up_indicator(security.symbol, security.indicator)

    # Remove the automatic indicator updates when the security leaves the universe.
    for security in changes.removed_securities:
        self.deregister_indicator(security.indicator)

Select Current Constituents

If you don't want to make any changes to the current universe, return Universe.UNCHANGED from your filter functions.

Select Language:
class MyUniverseAlgorithm(QCAlgorithm):
    def initialize(self) -> None:
        self.universe_settings.asynchronous = True
        self.add_universe(self._my_fundamental_filter_function)

    # Use Universe.UNCHANGED to leave the universe unchanged.
    def _my_fundamental_filter_function(self, fundamental: List[Fundamental]) -> List[Symbol]:
        return Universe.UNCHANGED

Universe Manager

The universe manager tracks all the universes in your algorithm. If you add multiple universe, you can access the constituents of each individual universe. To access the constituents of a universe in a multi-universe algorithm, save references to each universe when you add them.

Select Language:
# In the initialize method, add a new universe.
self.universe_settings.asynchronous = True
self._universe = self.add_universe(self._my_fundamental_filter_function)

# Use the universe_manager to get information about the assets in each universe.
universe_members = self.universe_manager[self._universe.configuration.symbol].members
for kvp in universe_members:
    symbol = kvp.key # The symbol of the asset.
    security = kvp.value # The security in the universe.

When you remove an asset from a universe, LEAN usually removes the security from the Members collection and removes the security subscription. However, it won't remove the security in any of the following situations:

  • You own the security.
  • You have an open order for the security.
  • The security wasn't in the universe long enough to meet the minimum_time_in_universe setting.

To get only the assets that are currently in the universe, see Selected Securities.

Active Securities

The QCAlgorithm.active_securities is a collection of Universe.members of all universes. The active_securities property of the algorithm class contains all of the assets currently in your algorithm. It is a dictionary where the key is a Symbol and the value is a Security. When you remove an asset from a universe, LEAN usually removes the security from the active_securities collection and removes the security subscription. However, it won't remove the security in any of the following situations:

  • You own the security.
  • You have an open order for the security.
  • The security wasn't in the universe long enough to meet the minimum_time_in_universe setting.

When LEAN removes the security, the Security object remains in the securities collection for record-keeping purposes, like tracking fees and trading volume.

To improve iteration speed, LEAN removes delisted securities from the securities primary collection. To access all the securities, iterate the securities.total property.

Select Language:
# Access all the securities.
for security in self.securities.total:
    pass

# Exclude delisted securities.
for security in self.securities.values():
    pass

To get only the assets that are currently in the universe, see Selected Securities.

Selected Securities

The selected property of your Universe contains references to all the assets that are currently in the universe. The Universe.selected property differs from the Universe.members property because the Universe.members property can contain more assets than Universe.selected when they have assets with open positions, open order orders, or assets that haven't met the minimum time requirement. To access the Universe object, save a reference to the result of the add_universe method. The following algorithm demonstrates how to use the Universe.selected property to create simple rebalancing strategies:

Select Language:
# Rebalance after market open to hold the 5 most liquid Equities.
class SimpleRebalancingAlgorithm(QCAlgorithm):

    def initialize(self):
        symbol = Symbol.create("SPY", SecurityType.EQUITY, Market.USA)
        # Schedule the universe selection to run at the start of each week.
        date_rule = self.date_rules.week_start(symbol)
        self.universe_settings.schedule.on(date_rule)
        # Add a universe of the top 5 most liquid Equities.
        universe = self.add_universe(self.universe.dollar_volume.top(5))
        # 30 minutes after market open, rebalance the portfolio.
        self.schedule.on(
            date_rule,
            self.time_rules.after_market_open(symbol, 30),
            lambda: self.set_holdings(
                [PortfolioTarget(symbol, 1/len(universe.selected)) for symbol in universe.selected], 
                True
            )
        )

Derivative Universes

In a regular universe, you select a basket of assets from the entire universe of securities. In a derivative universe, you select a basket of contracts for an underlying asset. The following derivative universes are available:

Historical Asset Prices

LEAN supports over 100 indicators you can use to generate your trading signals. However, in some cases, it's more efficient to maintain a matrix of historical prices for all the assets in your universe instead of using indicators. For example, say you want to calculate the correlation coefficients between all n asset in your universe using a 1-year lookback period. If you were to use the Correlation indicator, when a new asset enters the universe, you would usually need to make a 1-year history request for n assets to warm-up the n new correlation indicators. These large and frequent history requests can make your algorithm very slow.

The following algorithm demonstrates an alternative, more efficient approach. It defines a DataFrame in the initialize method. When a new asset enters the universe, it adds a column of historical prices for that asset to the DataFrame. Then, each day, it adds a single row of daily prices to the DataFrame, which covers all the assets in the universe. This approach reduces the size and frequency of history requests you need to make to get the data.

Select Language:
class MaintainHistoricalDailyUniversePriceDataAlgorithm(QCAlgorithm):

    def initialize(self):
        self.set_start_date(2010, 1, 1)
        self.set_cash(1_000_000)

        # Add a universe of daily data.
        self.universe_settings.resolution = Resolution.DAILY
        self._universe = self.add_universe(
            lambda fundamentals: [f.symbol for f in sorted(fundamentals, key=lambda f: f.market_cap)[-10:]]
        )

        # Create a DataFrame to store the historical data.
        self._all_history = pd.DataFrame()
        # Define the lookback period.
        self._lookback = 252  # Trading days.

        # Create a Scheduled Event to record new daily prices and
        # rebelance the portfolio.
        spy = Symbol.create('SPY', SecurityType.EQUITY, Market.USA)
        self.schedule.on(
            self.date_rules.every_day(spy),
            self.time_rules.at(0, 1), # One minute after `on_securities_changed` runs (in backtests)
            self._rebalance
        )

    def on_securities_changed(self, changes):
        # Remove the historical prices of assets that leave the universe.
        for security in changes.removed_securities:
            if security.symbol in self._all_history.columns:
                self._all_history.drop(security.symbol, axis=1, inplace=True)

        # Warm-up the historical data of assets that enter the universe.
        history = self.history(
            [security.symbol for security in changes.added_securities], self._lookback+1, Resolution.DAILY
        )
        if not history.empty:
            # If you trade at market open, it might make more sense to generate signals
            # based on daily opening prices. Drop the last row in `history` because 
            # `_rebalance` will add it.
            self._all_history = self._all_history.join(history.open.unstack(0).iloc[:-1], how='outer')

    def _rebalance(self):
        # Add yesterday's open price to the DataFrame of historical prices.
        self._all_history = pd.concat([
            self._all_history, 
            self.history(list(self._universe.selected), 1, Resolution.DAILY).open.unstack(0)
        ])
        # Trim the history to the lookback window size.
        self._all_history = self._all_history.iloc[-self._lookback:]

        # To avoid trading errors, select the assets that have a price.
        history = self._all_history[[symbol for symbol in self._all_history.columns if self.securities[symbol].price]]

        # Calculate asset signals for this rebalance.
        # For example, set the signal to give greater weight to uncorrelated assets.
        signal_by_symbol = 1/history.dropna(axis=1).corr().abs().sum()
        signal_by_symbol /= signal_by_symbol.sum()

        # Rebalance the portfolio based on the signals.
        self.set_holdings([PortfolioTarget(symbol, signal) for symbol, signal in signal_by_symbol.items()], True)

Examples

You may refer to the following table for examples of various asset classes:

Asset ClassUniverse TypeExample
EquitiesLiquidity UniversesExample
EquitiesFundamental UniversesExample
EquitiesETF Constituents UniversesExample
EquitiesChained UniversesExample
EquitiesAlternative Data UniversesExample
Equity OptionsEquity Option UniversesExample
CFDsEquity CFD UniversesExample
FuturesFuture UniversesExample
Future OptionsFuture Option UniversesExample
Index OptionsIndex Option UniversesExample
CryptosCrypto UniversesExample
AnyCustom UniversesExample
AnyDataless Scheduled UniversesExample

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: