Universes
Key Concepts
Introduction
Universe selection is the process of selecting a basket of assets you may trade. Dynamic universe selection increase diversification and decrease selection bias in your algorithm.
How Universe Selection Works
When you add a universe to your algorithm, LEAN sends a large dataset into a filter function you define. Your filter function needs to return a list of Symbol objects. LEAN automatically subscribes to these new symbols and adds them to your algorithm. Your algorithm can do almost anything inside your filter functions, but the goal should be to narrow down the set of securities to the securities that are most applicable for your algorithm.
Selection Functions
The following example filter function selects the 100 most liquid US Equities:
// Use a fundamental filter function to select the 100 most liquid Equities.
public class MyFundamentalUniverseAlgorithm : QCAlgorithm
{
public override void Initialize()
{
UniverseSettings.Asynchronous = true;
AddUniverse(FundamentalFilterFunction);
}
private IEnumerable<Symbol> FundamentalFilterFunction(IEnumerable<Fundamental> fundamental)
{
return (from c in fundamental
orderby c.DollarVolume descending
select c.Symbol).Take(100);
}
} # Use a fundamental filter function to select the 100 most liquid Equities.
class MyFundamentalUniverseAlgorithm(QCAlgorithm):
def initialize(self) -> None:
self.universe_settings.asynchronous = True
self.add_universe(self._fundamental_filter_function)
def _fundamental_filter_function(self, fundamental: List[Fundamental]) -> List[Symbol]:
sorted_by_dollar_volume = sorted(fundamental, key=lambda x: x.dollar_volume, reverse=True)
return [c.symbol for c in sorted_by_dollar_volume[:100]]
To learn how to define filter functions for other asset classes, custom data universes, or alternative data universes, see the following pages in this Universes chapter.
Security Changed Events
When your universe adds and removes assets, LEAN notifies your algorithm through the OnSecuritiesChangedon_securities_changed event handler. The event handler receives a SecurityChanges object, which contains references to the added and removed securities. To access the added securities, check the changes.AddedSecuritieschanges.added_securities method property. To access the removed securities, check the changes.RemovedSecuritieschanges.removed_securities method property.
// Use the OnSecuritiesChanged method to know when an asset is added or removed.
public override void OnSecuritiesChanged(SecurityChanges changes)
{
// Iterate through the added securities.
foreach (var security in changes.AddedSecurities)
{
Debug($"{Time}: Added {security.Symbol}");
}
// Iterate through the removed securities.
foreach (var security in changes.RemovedSecurities)
{
Debug($"{Time}: Removed {security.Symbol}");
if (security.Invested)
{
Liquidate(security.Symbol, "Removed from Universe");
}
}
} # Use the on_securities_changed method to know when an asset is added or removed.
def on_securities_changed(self, changes: SecurityChanges) -> None:
# Iterate through the added securities.
for security in changes.added_securities:
self.debug(f"{self.time}: Added {security.symbol}")
# Iterate through the removed securities.
for security in changes.removed_securities:
self.debug(f"{self.time}: Removed {security.symbol}")
if security.invested:
self.liquidate(security.symbol, "Removed from Universe")
The preceding example liquidates securities that leave the universe. In this case, LEAN creates a market on open order and frees up buying power when the market opens.
Custom Security Properties
If you need to save data or create objects for each security in the universe, add custom members to the respective Security objectscast the Security objects to dynamic objects and then save custom members to them. This technique is useful if you want to track stop loss levels or add indicators for each asset in the universe.
// Cast the security to a dynamic object to add custom properties to it.
public override void OnSecuritiesChanged(SecurityChanges changes)
{
foreach (var security in changes.AddedSecurities)
{
var dynamicSecurity = security as dynamic;
// Create an SMA indicator with 10 periods for the asset.
dynamicSecurity.Indicator = SMA(security.Symbol, 10);
// Warm up the indicator with historical data.
WarmUpIndicator(security.Symbol, dynamicSecurity.Indicator);
}
// Remove the automatic indicator updates when the security leaves the universe.
foreach (var security in changes.RemovedSecurities)
{
DeregisterIndicator((security as dynamic).Indicator);
}
} # Add custom properties to the security by duck-typing.
def on_securities_changed(self, changes: SecurityChanges) -> None:
for security in changes.added_securities:
# Create an SMA indicator with 10 periods for the asset.
# Use duck typing to store it on the security object.
security.indicator = self.sma(security.Symbol, 10)
# Warm up the indicator with historical data.
self.warm_up_indicator(security.symbol, security.indicator)
# Remove the automatic indicator updates when the security leaves the universe.
for security in changes.removed_securities:
self.deregister_indicator(security.indicator)
Select Current Constituents
If you don't want to make any changes to the current universe, return Universe.UnchangedUniverse.UNCHANGED from your filter functions.
public class MyUniverseAlgorithm : QCAlgorithm
{
public override void Initialize()
{
UniverseSettings.Asynchronous = true;
AddUniverse(MyFundamentalFilterFunction);
}
// Use Universe.Unchanged to leave the universe unchanged.
IEnumerable<Symbol> MyFundamentalFilterFunction(IEnumerable<Fundamental> fundamental)
{
return Universe.Unchanged;
}
} class MyUniverseAlgorithm(QCAlgorithm):
def initialize(self) -> None:
self.universe_settings.asynchronous = True
self.add_universe(self._my_fundamental_filter_function)
# Use Universe.UNCHANGED to leave the universe unchanged.
def _my_fundamental_filter_function(self, fundamental: List[Fundamental]) -> List[Symbol]:
return Universe.UNCHANGED
Universe Manager
The universe manager tracks all the universes in your algorithm. If you add multiple universe, you can access the constituents of each individual universe. To access the constituents of a universe in a multi-universe algorithm, save references to each universe when you add them.
private Universe _universe;
// In the Initialize method, add a new universe.
UniverseSettings.Asynchronous = true;
_universe = AddUniverse(MyFundamentalFilterFunction);
// Use the UniverseManager to get information about the assets in each universe.
var universeMembers = UniverseManager[_universe.Configuration.Symbol].Members;
foreach (var kvp in universeMembers)
{
var symbol = kvp.Key; // The symbol of the asset.
var security = kvp.Value; // The security in the universe.
} # In the initialize method, add a new universe.
self.universe_settings.asynchronous = True
self._universe = self.add_universe(self._my_fundamental_filter_function)
# Use the universe_manager to get information about the assets in each universe.
universe_members = self.universe_manager[self._universe.configuration.symbol].members
for kvp in universe_members:
symbol = kvp.key # The symbol of the asset.
security = kvp.value # The security in the universe.
When you remove an asset from a universe, LEAN usually removes the security from the Members collection and removes the security subscription.
However, it won't remove the security in any of the following situations:
- You own the security.
- You have an open order for the security.
- The security wasn't in the universe long enough to meet the
MinimumTimeInUniverseminimum_time_in_universesetting.
To get only the assets that are currently in the universe, see Selected Securities.
Active Securities
The QCAlgorithm.ActiveSecuritiesQCAlgorithm.active_securities is a collection of Universe.MembersUniverse.members of all universes. The ActiveSecuritiesactive_securities property of the algorithm class contains all of the assets currently in your algorithm. It is a dictionary where the key is a Symbol and the value is a Security. When you remove an asset from a universe, LEAN usually removes the security from the ActiveSecuritiesactive_securities collection and removes the security subscription. However, it won't remove the security in any of the following situations:
- You own the security.
- You have an open order for the security.
- The security wasn't in the universe long enough to meet the
MinimumTimeInUniverseminimum_time_in_universesetting.
When LEAN removes the security, the Security object remains in the Securitiessecurities collection for record-keeping purposes, like tracking fees and trading volume.
To improve iteration speed, LEAN removes delisted securities from the Securitiessecurities primary collection. To access all the securities, iterate the Securities.Totalsecurities.total property.
// Access all the securities.
foreach (var security in Securities.Total)
{
}
// Exclude delisted securities.
foreach (var security in Securities.Values)
{
} # Access all the securities.
for security in self.securities.total:
pass
# Exclude delisted securities.
for security in self.securities.values():
pass
To get only the assets that are currently in the universe, see Selected Securities.
Selected Securities
The Selectedselected property of your Universe contains references to all the assets that are currently in the universe.
The Universe.SelectedUniverse.selected property differs from the Universe.MembersUniverse.members property because the Universe.MembersUniverse.members property can contain more assets than Universe.SelectedUniverse.selected when they have assets with open positions, open order orders, or assets that haven't met the minimum time requirement.
To access the Universe object, save a reference to the result of the AddUniverseadd_universe method.
The following algorithm demonstrates how to use the Universe.SelectedUniverse.selected property to create simple rebalancing strategies:
// Rebalance after market open to hold the 5 most liquid Equities.
public class SimpleRebalancingAlgorithm : QCAlgorithm
{
public override void Initialize()
{
var symbol = QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA);
// Schedule the universe selection to run at the start of each week.
var dateRule = DateRules.WeekStart(symbol);
UniverseSettings.Schedule.On(dateRule);
// Add a universe of the top 5 most liquid Equities.
var universe = AddUniverse(Universe.DollarVolume.Top(5));
// 30 minutes after market open, rebalance the portfolio.
Schedule.On(
dateRule,
TimeRules.AfterMarketOpen(symbol, 30),
() => SetHoldings(
universe.Selected.Select(symbol => new PortfolioTarget(symbol, 1.0m/universe.Selected.Count)).ToList(),
true
)
);
}
} # Rebalance after market open to hold the 5 most liquid Equities.
class SimpleRebalancingAlgorithm(QCAlgorithm):
def initialize(self):
symbol = Symbol.create("SPY", SecurityType.EQUITY, Market.USA)
# Schedule the universe selection to run at the start of each week.
date_rule = self.date_rules.week_start(symbol)
self.universe_settings.schedule.on(date_rule)
# Add a universe of the top 5 most liquid Equities.
universe = self.add_universe(self.universe.dollar_volume.top(5))
# 30 minutes after market open, rebalance the portfolio.
self.schedule.on(
date_rule,
self.time_rules.after_market_open(symbol, 30),
lambda: self.set_holdings(
[PortfolioTarget(symbol, 1/len(universe.selected)) for symbol in universe.selected],
True
)
)
Historical Asset Prices
LEAN supports over 100 indicators you can use to generate your trading signals. However, in some cases, it's more efficient to maintain a matrix of historical prices for all the assets in your universe instead of using indicators. For example, say you want to calculate the correlation coefficients between all $n$ asset in your universe using a 1-year lookback period. If you were to use the Correlation indicator, when a new asset enters the universe, you would usually need to make a 1-year history request for $n$ assets to warm-up the $n$ new correlation indicators. These large and frequent history requests can make your algorithm very slow.
The following algorithm demonstrates an alternative, more efficient approach.
It defines a Dictionary when constructing the algorithm classDataFrame in the initialize method.
When a new asset enters the universe, it adds a column of historical prices for that asset to the DataFrame.
Then, each day, it adds a single row of daily prices to the DataFrame, which covers all the assets in the universe.
When a new asset enters the universe, it adds a new key-value pair to the dictionary.
Then, each day, it adds yesterday's open price to each RollingWindow in the dictionary, which covers all the assets in the universe.
This approach reduces the size and frequency of history requests you need to make to get the data.
using Accord.Statistics;
public class MaintainHistoricalDailyUniversePriceDataAlgorithm : QCAlgorithm
{
// Create Rolling Window collection to store the historical data.
private Dictionary<Symbol, RollingWindow<decimal>> _history = new();
// Define the lookback period.
private int _lookback = 252;
private Universe _universe;
public override void Initialize()
{
SetStartDate(2010, 1, 1);
SetCash(1000000);
// Add a universe of daily data.
UniverseSettings.Resolution = Resolution.Daily;
// Filter for any coporate announced a material buyback plan, since they have confidence in their
// future prospect and the reduction in supply can drive their price up.
_universe = AddUniverse((fundamentals) => {
return (from Fundamental f in fundamentals
orderby f.MarketCap descending
select f.Symbol).Take(10);
});
// Create a Scheduled Event to record new daily prices and rebelance the portfolio.
var spy = QuantConnect.Symbol.Create("SPY", SecurityType.Equity, Market.USA);
Schedule.On(
DateRules.EveryDay(spy),
TimeRules.At(0, 1), // One minute after `OnSecuritiesChanged` runs (in backtests)
Rebalance
);
}
public override void OnSecuritiesChanged(SecurityChanges changes)
{
// Remove the historical prices of assets that leave the universe.
foreach (var removed in changes.RemovedSecurities)
{
_history.Remove(removed.Symbol);
}
// Warm-up the historical data of assets that enter the universe.
var symbols = changes.AddedSecurities.Select(x => x.Symbol).ToList();
var history = History<TradeBar>(symbols, _lookback+1, Resolution.Daily);
foreach (var bars in history.SkipLast(1))
{
foreach (var (symbol, bar) in bars)
{
if (!_history.TryGetValue(symbol, out var symbolHistory))
{
_history[symbol] = symbolHistory = new(_lookback);
}
// If you trade at market open, it might make more sense to generate signals based on daily
// opening prices.
symbolHistory.Add(bar.Open);
}
}
}
private void Rebalance()
{
// Add yesterday's open price to the Dictionary of historical prices.
var history = History<TradeBar>(_history.Keys, 1, Resolution.Daily).First();
foreach (var (symbol, bar) in history)
{
_history[symbol].Add(bar.Open);
}
// To avoid trading errors, select the assets that have a price.
var symbols = _history.Keys.Where(symbol => Securities[symbol].Price > 0).ToArray();
// Calculate asset signals for this rebalance.
// For example, set the signal to give greater weight to uncorrelated assets.
var priceMatrix = new double[_lookback, symbols.Count()];
for (int i = 0; i < _lookback; i++)
{
for (int j = 0; j < symbols.Count(); j++)
{
priceMatrix[i, j] = (double)_history[symbols[j]][i];
}
}
var correlation = Measures.Correlation(priceMatrix);
var signals = NormalizedRowAbsoluteSums(correlation);
// Rebalance the portfolio based on the signals.
SetHoldings(
Enumerable
.Range(0, signals.Length)
.Select(i => new PortfolioTarget(symbols[i], Convert.ToDecimal(signals[i])))
.ToList(),
true
);
}
private double[] NormalizedRowAbsoluteSums(double[,] array)
{
var rows = array.GetLength(0);
var rowAbsSums = new double[rows];
for (int i = 0; i < rows; i++)
{
var rowAbsSum = 0d;
for (int j = 0; j < array.GetLength(1); j++)
{
rowAbsSum += Math.Abs(array[i, j]);
}
rowAbsSums[i] = rowAbsSum;
}
var inverseRowSums = rowAbsSums.Select(sum => 1.0d/sum);
var allSum = inverseRowSums.Sum();
return inverseRowSums.Select(x => x / allSum).ToArray();
}
} class MaintainHistoricalDailyUniversePriceDataAlgorithm(QCAlgorithm):
def initialize(self):
self.set_start_date(2010, 1, 1)
self.set_cash(1_000_000)
# Add a universe of daily data.
self.universe_settings.resolution = Resolution.DAILY
self._universe = self.add_universe(
lambda fundamentals: [f.symbol for f in sorted(fundamentals, key=lambda f: f.market_cap)[-10:]]
)
# Create a DataFrame to store the historical data.
self._all_history = pd.DataFrame()
# Define the lookback period.
self._lookback = 252 # Trading days.
# Create a Scheduled Event to record new daily prices and
# rebelance the portfolio.
spy = Symbol.create('SPY', SecurityType.EQUITY, Market.USA)
self.schedule.on(
self.date_rules.every_day(spy),
self.time_rules.at(0, 1), # One minute after `on_securities_changed` runs (in backtests)
self._rebalance
)
def on_securities_changed(self, changes):
# Remove the historical prices of assets that leave the universe.
for security in changes.removed_securities:
if security.symbol in self._all_history.columns:
self._all_history.drop(security.symbol, axis=1, inplace=True)
# Warm-up the historical data of assets that enter the universe.
history = self.history(
[security.symbol for security in changes.added_securities], self._lookback+1, Resolution.DAILY
)
if not history.empty:
# If you trade at market open, it might make more sense to generate signals
# based on daily opening prices. Drop the last row in `history` because
# `_rebalance` will add it.
self._all_history = self._all_history.join(history.open.unstack(0).iloc[:-1], how='outer')
def _rebalance(self):
# Add yesterday's open price to the DataFrame of historical prices.
self._all_history = pd.concat([
self._all_history,
self.history(list(self._universe.selected), 1, Resolution.DAILY).open.unstack(0)
])
# Trim the history to the lookback window size.
self._all_history = self._all_history.iloc[-self._lookback:]
# To avoid trading errors, select the assets that have a price.
history = self._all_history[[symbol for symbol in self._all_history.columns if self.securities[symbol].price]]
# Calculate asset signals for this rebalance.
# For example, set the signal to give greater weight to uncorrelated assets.
signal_by_symbol = 1/history.dropna(axis=1).corr().abs().sum()
signal_by_symbol /= signal_by_symbol.sum()
# Rebalance the portfolio based on the signals.
self.set_holdings([PortfolioTarget(symbol, signal) for symbol, signal in signal_by_symbol.items()], True)
You can also see our Videos. You can also get in touch with us via Discord.
Did you find this page helpful?
