book
Checkout our new book! Hands on AI Trading with Python, QuantConnect, and AWS Learn More arrow

Historical Data

Universe Data

Introduction

Universe selection is the process of selecting a basket of assets you may trade. Dynamic universe selection increase diversification and decrease selection bias in your algorithm. To get the history of a universe, call the history method with the universe object. Set the flatten argument to True to organize the data in a DataFrame for easy pandas wrangling. Note that the history method doesn't filter the universe with your filter function. It simply returns all of the assets and data points in the universe dataset.

US Equity Fundamentals

The following example gets the history of a US Equity fundamental universe:

Select Language:
class USEquityFundamentalUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 23)
        # Add a universe of US Equities based on the Fundamentals dataset.
        universe = self.add_universe(lambda fundamentals: [f.symbol for f in fundamentals])
        # Get 5 days of history for the universe.
        history = self.history(universe, timedelta(5), flatten=True)
adjustedpriceassetclassificationcompanyprofilecompanyreferencedollarvolumeearningratiosearningreportsfinancialstatementshasfundamentaldatamarketmarketcapoperationratiospricefactorpricescalefactorsecurityreferencesplitfactorvaluationratiosvaluevolume
timesymbol
2024-12-19A RPTMYV3VC57P132.764597QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference219150734.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa39348079727QuantConnect.Data.Fundamental.OperationRatios0.9981550.998155QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios133.01001647626
AA R735QTJ8XC9X107.606519QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference262034107.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa48093135628QuantConnect.Data.Fundamental.OperationRatios0.9992250.999225QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios107.69002433226
AA WF7IHVI76I5H37.310000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference159500884.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa11995414664QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios37.31004275017
AAA XHPVXIJUWS4L24.822618QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference494441.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsFalseusa0QuantConnect.Data.Fundamental.OperationRatios0.9869470.986947QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios25.150919659
AAAU WX1HYU092ESL25.635000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference57235085.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsFalseusa0QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios25.63502232693
...............................................................
2024-12-21ZUO WTMF35A5I3QD9.910000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference76550429.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa1526407764QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios9.91007724564
ZVIA XQD1725736UD3.180000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference1620426.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa91666688QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios3.1800509568
ZVSA Y4A7QNC417J91.140000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference245696.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa2648935QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios1.1400215523
ZYME WK2S2IM8SSPX14.120000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference10554318.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa1075178180QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios14.1200747473
ZYXI X1ZPVR7SWYUD8.250000QuantConnect.Data.Fundamental.AssetClassificationQuantConnect.Data.Fundamental.CompanyProfileQuantConnect.Data.Fundamental.CompanyReference2580509.0QuantConnect.Data.Fundamental.EarningRatiosQuantConnect.Data.Fundamental.EarningReportsQuantConnect.Data.Fundamental.FinancialStatementsTrueusa296163568QuantConnect.Data.Fundamental.OperationRatios1.0000001.000000QuantConnect.Data.Fundamental.SecurityReference1.0QuantConnect.Data.Fundamental.ValuationRatios8.2500312789
# Get the PE Ratio of the first row in the DataFrame.
pe_ratio = history.iloc[0].valuationratios.pe_ratio

US Equity ETF Constituents

The following example gets the history of a US Equity ETF constituents universe:

Select Language:
class ETFConstituentUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 23)
        # Add a universe of US Equities based on the constituents of an ETF.
        universe = self.add_universe(self.universe.etf('SPY'))
        # Get 5 days of history for the universe.
        history = self.history(universe, timedelta(5), flatten=True)
lastupdateperiodsharesheldweight
timesymbol
2024-12-19A RPTMYV3VC57P2024-12-171 days3682327.00.000777
AAPL R735QTJ8XC9X2024-12-171 days191633535.00.075065
ABBV VCY032R250MD2024-12-171 days22256743.00.006032
ABNB XK8H247DY6W52024-12-171 days5536841.00.001131
ABT R735QTJ8XC9X2024-12-171 days21919703.00.003837
..................
2024-12-21XYL V18KR26TE3XH2024-12-191 days2947931.00.000569
YUM R735QTJ8XC9X2024-12-191 days3412693.00.000737
ZMH S6ZZPKTVDY052024-12-191 days2471688.00.000433
ZBRA R735QTJ8XC9X2024-12-191 days625772.00.000400
ZTS VDRJHVQ4FNFP2024-12-191 days5484351.00.001486
# Select the 2 assets with the smallest weights in the ETF each day.
daily_smallest = history.groupby('time').apply(lambda x: x.nsmallest(2, 'weight')).reset_index(level=1, drop=True).weight
time        symbol            
2024-12-19  AMTMW YM37RIGZUD0L    0.000053
            NWSVV VHJF6S7EZRL1    0.000068
2024-12-20  AMTMW YM37RIGZUD0L    0.000051
            NWSVV VHJF6S7EZRL1    0.000069
2024-12-21  AMTMW YM37RIGZUD0L    0.000048
            NWSVV VHJF6S7EZRL1    0.000069
Name: weight, dtype: float64

US Equity Alternative Data

The following example gets the history of a US Equity alternative data universe:

Select Language:
class USEquityAltDataUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 23)
        # Add a universe of US Equities based on an alternative dataset.
        universe = self.add_universe(BrainStockRankingUniverse)
        # Get 5 days of history for the universe.
        history = self.history(universe, timedelta(5), flatten=True)
rank10daysrank21daysrank2daysrank3daysrank5daysvalue
timesymbol
2024-12-18A RPTMYV3VC57P-0.0018950.005938-0.007858-0.0063200.001771-0.007858
AAL VM9RIYHM8ACL-0.0039770.006520-0.005671-0.003738-0.004715-0.005671
AAPL R735QTJ8XC9X0.0274500.0373390.0060180.0014890.0101020.006018
ABBV VCY032R250MD-0.0028140.012297-0.001717-0.000679-0.007454-0.001717
ABNB XK8H247DY6W50.0205330.046173-0.0023030.0073500.011252-0.002303
........................
2024-12-23ZI XF2DKG2HHLK5-0.038118-0.037527-0.035904-0.041195-0.041388-0.035904
ZM X3RPXTZRW09X0.0067840.020690-0.005674-0.008556-0.002120-0.005674
ZS WSVU0MELFQED0.0104220.019619-0.003743-0.0020790.002778-0.003743
ZTO WF2L9EOCSQCL-0.021455-0.015939-0.018053-0.020263-0.024979-0.018053
ZTS VDRJHVQ4FNFP0.0305890.0414000.0247440.0198910.0310480.024744
# Select the asset with the greatest value each day.
daily_winner = history.groupby('time').apply(lambda x: x.nlargest(1, 'value')).reset_index(level=1, drop=True).value
time        symbol          
2024-12-18  FIC R735QTJ8XC9X    0.054204
2024-12-19  FIC R735QTJ8XC9X    0.073250
2024-12-20  FIC R735QTJ8XC9X    0.065142
2024-12-21  FIC R735QTJ8XC9X    0.065142
2024-12-22  FIC R735QTJ8XC9X    0.065142
2024-12-23  FIC R735QTJ8XC9X    0.065142
Name: value, dtype: float64

Equity Options

The following example gets the history of an Equity Options universe:

Select Language:
class EquityOptionUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 23)
        # Add a universe of Equity Option contracts.
        option = self.add_option('SPY')
        # Get 5 days of history for the universe.
        history = self.history(option.symbol, timedelta(5), flatten=True)
closedeltagammahighimpliedvolatilitylowopenopeninterestrhothetaunderlyingvaluevegavolume
timesymbol
2024-12-19SPY YOGVNNCO8QDI|SPY R735QTJ8XC9X466.3500.0000000.000000486.9100.000000465.655484.0001543.00.0000000.000000SPY: ¤586.28466.3500.0000007.0
SPY YOGVNNCU72FA|SPY R735QTJ8XC9X455.8000.0000000.000000476.9100.000000455.575474.01031.00.009492-0.000385SPY: ¤586.28455.800-0.0016215.0
SPY YOGVNND05EH2|SPY R735QTJ8XC9X445.9600.0000000.000000466.9100.000000445.750463.9859.00.010222-0.001891SPY: ¤586.28445.960-0.0056385.0
SPY YOGVNND63QIU|SPY R735QTJ8XC9X436.4450.0000000.000000456.9150.000000435.545454.00523.00.010952-0.003397SPY: ¤586.28436.445-0.0096565.0
SPY YTG30NXW11QE|SPY R735QTJ8XC9X438.3801.0000000.000000458.4750.722009437.950456.65033.00.0000000.000000SPY: ¤586.28438.3800.0000000.0
................................................
2024-12-21SPY 33899RRUZK23Q|SPY R735QTJ8XC9X309.500-0.8670020.000479357.3800.141164272.505318.2201.0-5.7586720.037828SPY: ¤591.15309.5000.5494880.0
SPY 337HP99QFUJJA|SPY R735QTJ8XC9X314.500-0.8776580.000388362.4250.136831277.490323.1900.0-5.6595640.039793SPY: ¤591.15314.5000.4147550.0
SPY 33899RVZ5ZO12|SPY R735QTJ8XC9X314.315-0.8707420.000427362.3700.138433277.495322.9400.0-5.8458950.038531SPY: ¤591.15314.3150.4818760.0
SPY 337HP95ZTK8HY|SPY R735QTJ8XC9X319.500-0.8778730.000381367.4250.138304282.490328.1900.0-5.7495420.040433SPY: ¤591.15319.5000.4116210.0
SPY 33899RS8JPCZQ|SPY R735QTJ8XC9X319.500-0.8698150.000438367.3700.141717282.495328.2001.0-5.9413540.039119SPY: ¤591.15319.5000.5032190.0
# Find the open interest value at the 99th percentile.
high_open_interest = history.openinterest.quantile(0.99)

Index Options

The following example gets the history of an Index Options universe:

Select Language:
class IndexOptionsUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 23)
        # Add a universe of Index Option contracts.
        option = self.add_index_option('VIX')
        # Get 5 days of history for the universe.
        history = self.history(option.symbol, timedelta(5), flatten=True)
closedeltagammahighimpliedvolatilitylowopenopeninterestrhothetaunderlyingvaluevegavolume
timesymbol
2024-12-19VIX YPDGYCCGNK32|VIX 3111.5001.0000000.00000011.9500.0000006.4256.5753868.00.008740-0.001349VIX: ¤27.6511.5000.0000003747.0
VIX YQ51GYL6SOKU|VIX 3111.1001.0000000.00000011.8250.0000008.2258.55059.00.015541-0.001343VIX: ¤27.6511.1000.0000009.0
VIX YPDGYCB530UM|VIX 3110.5001.0000000.00000011.0500.0000004.2556.805905.00.009711-0.001499VIX: ¤27.6510.5000.000000250.0
VIX YQ51GYJV85CE|VIX 3110.1251.0000000.00000010.3250.0000007.2257.550225.00.017268-0.001492VIX: ¤27.6510.1250.000000112.0
VIX YQVMJHFFAIOE|VIX 319.9001.0000000.00000010.2250.0000005.4305.580192.00.024495-0.001486VIX: ¤27.659.9000.00000022.0
................................................
2024-12-21VIX 32P91MEJ8IFXQ|VIX 31179.925-0.9874820.001990182.1252.167690176.675177.7001.0-0.3308960.025512VIX: ¤18.37179.9250.0024330.0
VIX 32PZMOXESKT9Q|VIX 31179.150-0.9846470.002316180.8501.849262176.050177.35084.0-0.4753290.026046VIX: ¤18.37179.1500.0034850.0
VIX 32QS6NN0OSOXA|VIX 31178.425-0.9819290.002610180.1001.636117174.450176.87520.0-0.6291000.026343VIX: ¤18.37178.4250.0046200.0
VIX 32RQNAWTLH3JI|VIX 31177.500-0.9818960.002619179.3001.433080174.200176.2253.0-0.8129860.026931VIX: ¤18.37177.5000.0052740.0
VIX 32SI7TJ2BM81A|VIX 31176.725-0.9827930.002528178.5751.305996173.400175.6004.0-0.9587440.027301VIX: ¤18.37176.7250.0054950.0
# Get the daily trading volume across all the contracts.
daily_volume = history.groupby('time').apply(lambda x: x.volume.sum())
time
2024-12-19    1285753.0
2024-12-20    1374216.0
2024-12-21     833384.0
dtype: float64

Crypto

The following example gets the history of a Crypto universe:

Select Language:
class CryptoUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 23)
        # Add a universe of Cryptocurrencies.
        universe = self.add_universe(CryptoUniverse.coinbase(lambda data: [x.symbol for x in data]))
        # Get 5 days of history for the universe.
        history = self.history(universe, timedelta(5), flatten=True)
closehighlowopenpricevolumevolumeinusd
timesymbol
2024-12-1900USD 2XR0.0503000.0515000.0492000.0515000.0503003.173495e+061.596268e+05
00USDC 2XR0.0503000.0515000.0492000.0515000.0503003.173495e+061.596268e+05
1INCHBTC 2XR0.0000040.0000050.0000040.0000050.0000049.815380e+034.584371e+03
1INCHEUR 2XR0.4460000.4670000.4420000.4650000.4460009.007325e+044.215390e+04
1INCHGBP 2XR0.3690000.3850000.3660000.3810000.3690003.932220e+041.845807e+04
...........................
2024-12-23ZETAUSDC 2XR0.5745000.6386000.5626000.5984000.5745001.522481e+068.746654e+05
ZROUSD 2XR5.4870006.3630005.3930005.6350005.4870001.312069e+057.199321e+05
ZROUSDC 2XR5.4870006.3630005.3930005.6350005.4870001.312069e+057.199321e+05
ZRXUSD 2XR0.4499440.5094250.4415900.4760010.4499443.371118e+061.516814e+06
ZRXUSDC 2XR0.4499440.5094250.4415900.4760010.4499443.371118e+061.516814e+06
# Get the Crypto with the largest daily return each day.
history['roc'] = (history['close'] - history['open']) / history['open']
top_gainers = history.loc[history.groupby('time')['roc'].idxmax()]
closehighlowopenpricevolumevolumeinusdroc
timesymbol
2024-12-19PRQUSD 2XR0.190200.210000.149300.153600.190201.934896e+073.680171e+060.238281
2024-12-20NCTUSD 2XR0.036960.038440.027400.029560.036962.584655e+089.552883e+060.250338
2024-12-21VELOUSD 2XR0.220990.238630.190020.200750.220998.206130e+071.813473e+070.100822
2024-12-22ZENUSD 2XR26.1060028.3900015.7050015.8330026.106005.834163e+051.523067e+070.648835
2024-12-23AGLDUSD 2XR2.154402.170301.305401.310102.154405.144987e+061.108436e+070.644455

Crypto Market Cap

The following example gets the history of a Crypto market cap universe:

Select Language:
class CryptoMarketCapUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2024, 12, 24)
        # Add a universe of Cryptocurrencies.
        universe = self.add_universe(CoinGeckoUniverse)
        # Get 355 days of history for the universe.
        history = self.history(universe, timedelta(365), flatten=True)
coinmarketcapvaluevolume
timesymbol
2023-12-26DAPP.CoinGecko 2SDAPP0.000000e+000.0001111.203664e+03
FLUID.CoinGecko 2SFLUID3.385875e+071.8866538.124432e+04
PAAL.CoinGecko 2SPAAL7.678778e+070.1180193.264158e+06
TOSHI.CoinGecko 2STOSHI5.358157e+070.0001279.605914e+05
2023-12-27DAPP.CoinGecko 2SDAPP0.000000e+000.0001091.189876e+03
..................
2024-12-24VIRTUAL.CoinGecko 2SVIRTUAL3.016890e+093.0148024.121598e+08
XION.CoinGecko 2SXION7.990578e+073.1265831.454954e+07
ZAP.CoinGecko 2SZAP3.094296e+060.0326121.781765e+05
ZEREBRO.CoinGecko 2SZEREBRO3.018897e+080.3009907.376285e+07
ZRC.CoinGecko 2SZRC1.660151e+080.0756401.373892e+08
# Select the largest Crypto each day.
largest_coin = history.groupby('time').apply(
    lambda x: x.nlargest(1, 'marketcap')
).reset_index(level=1, drop=True).marketcap
time        symbol              
2023-12-26  PAAL.CoinGecko 2S       7.678778e+07
2023-12-27  PAAL.CoinGecko 2S       6.908865e+07
2023-12-28  PAAL.CoinGecko 2S       1.019541e+08
2023-12-29  PAAL.CoinGecko 2S       1.007006e+08
2023-12-30  PAAL.CoinGecko 2S       8.907086e+07
                                        ...     
2024-12-20  VIRTUAL.CoinGecko 2S    2.297529e+09
2024-12-21  VIRTUAL.CoinGecko 2S    2.811267e+09
2024-12-22  VIRTUAL.CoinGecko 2S    2.476485e+09
2024-12-23  VIRTUAL.CoinGecko 2S    2.328428e+09
2024-12-24  VIRTUAL.CoinGecko 2S    3.016890e+09
Name: marketcap, Length: 365, dtype: float64

Custom Data

The following example gets the history of a custom data universe:

Select Language:
# Define the custom data class outside of the algorithm class.    
class StockDataSource(PythonData):

    def get_source(self, config: SubscriptionDataConfig, date: datetime, is_live: bool) -> SubscriptionDataSource:
        return SubscriptionDataSource(
            "https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/csv-universe-example.csv", 
            SubscriptionTransportMedium.REMOTE_FILE, 
            FileFormat.CSV
        )

    def reader(self,config: SubscriptionDataConfig, line: str, date: datetime, is_live: bool) -> BaseData:
        if not (line.strip() and line[0].isdigit()): 
            return None
        stocks = StockDataSource()
        stocks.symbol = config.symbol
        try:
            csv = line.split(',')
            stocks.time = datetime.strptime(csv[0], "%Y%m%d")
            stocks.end_time = stocks.time + timedelta(days=1)
            stocks["Symbols"] = csv[1:]
        except ValueError:
            return None
        return stocks


class CustomDataUniverseHistoryAlgorithm(QCAlgorithm):

    def initialize(self) -> None:
        self.set_start_date(2018, 1, 10)
        # Add a custom data universe.
        universe = self.add_universe(StockDataSource)
        # Get the universe history.
        history = self.history(universe, datetime(2018, 1, 1), datetime(2018, 1, 10), flatten=True)
symbols
symboltime
STOCKDATASOURCE-USA-AE903F8D-F507-4638-BBFC-EE65760179AE.StockDataSource 2S2018-01-04[SPY, QQQ, AAPL, IWM, FB]
2018-01-05[AAPL, QQQ, IWM, FB, GOOGL]
2018-01-06[QQQ, IWM, FB, BAC, GOOGL]
2018-01-07[IWM, FB, GOOGL, BAC, GOOG]
2018-01-08[IWM, FB, GOOGL, BAC, GOOG]
2018-01-09[IWM, FB, GOOGL, BAC, GOOG]
2018-01-10[IWM, BAC, GOOGL, GOOG, IBM]
# Count the number of assets in the universe each day.
universe_size = history.apply(lambda row: len(row.symbols), axis=1).reset_index(level=0, drop=True)
time
2018-01-04    5
2018-01-05    5
2018-01-06    5
2018-01-07    5
2018-01-08    5
2018-01-09    5
2018-01-10    5
dtype: int64

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: