Security not found in self.ActiveSecurities after added through self.AddEquity

Hi Haakon,

Your way of doing Universe is rather unorthodox but well, I might post the more orthodox way to do custom universe later on.

For now, named parameters are not supported since Python in LEAN is only a Python.NET front-end to .NET code. So you need to change your calls to AddEquity, like so:

# region imports
from AlgorithmImports import *
# endregion

class Test(QCAlgorithm):

    def Initialize(self):
        self.SetStartDate(2022, 4, 11)  # Set Start Date
        self.SetEndDate(2022, 4, 13)  # Set End Date
        self.SetCash(100000)  # Set Strategy Cash

        # Raw data normalization
        self.SetSecurityInitializer(lambda x: x.SetDataNormalizationMode(DataNormalizationMode.Adjusted))

        # Add SPY and define symbols
        self.AddEquity("SPY", Resolution.Minute, Market.USA, True, Security.NullLeverage, True)
        self.selectedSymbols = []

        # Add universe
        self.AddUniverse(self.CoarseUniverseSelection)

        # Scheduled event
        self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.At(9, 25), self.CheckStocks)
        self.Schedule.On(self.DateRules.EveryDay("SPY"), self.TimeRules.At(21, 00), self.ResetUniverse)

    def OnData(self, data: Slice):
        pass

    def CoarseUniverseSelection(self, coarse):
        self.Debug(f"Universe selection time: {self.Time}")
        for stock in coarse:
            if (stock.Price > 10):
                # Subscribe to equity
                self.AddEquity(stock.Symbol, Resolution.Minute, Market.USA, True, Security.NullLeverage, True)
                self.selectedSymbols.append(stock.Symbol)
        
        return []

    
    def CheckStocks(self):
        self.Debug(f"Check stocks time: {self.Time}")
        number_not_in_active_securities = 0
        # Remove security
        for symbol in self.selectedSymbols:
            try:
                security = self.ActiveSecurities[symbol] # Can't find security even though it was added at the start of the day by CoarseUniverseSelection
                price = security.Price
            except:
                number_not_in_active_securities += 1
        
        self.Debug(f"{number_not_in_active_securities} of {len(self.selectedSymbols)} not in active securities")

    def ResetUniverse(self):
        self.Debug(f"Resetting universe: {self.Time}")
        self.selectedSymbols = []
        self.yesterdayClose = {}

        for kvp in self.ActiveSecurities:
            symbol = kvp.Key
            self.RemoveSecurity(symbol)
    
        self.AddEquity("SPY", Resolution.Minute, Market.USA, True, Security.NullLeverage, True)

Fred

Hi,

QuantConnect's version of pythonet (like the master version) supports names arguments, so 

self.AddEquity("SPY", resolution=Resolution.Minute, extendedMarketHours=True)

is supported.

I would recommend avoiding mixing Universe Selection with AddEquity:

    def CoarseUniverseSelection(self, coarse):
        self.Debug(f"Universe selection time: {self.Time}")
        for stock in coarse:
            if (stock.Price > 10):
                # Subscribe to equity
                self.AddEquity(stock.Symbol, resolution=Resolution.Minute, extendedMarketHours=True)
                self.selectedSymbols.append(stock.Symbol)
        
        return []

and follow the documentation on the topic

    def Initialize(self):
    	self.UniverseSettings.ExtendedMarketHours = True
          
    def CoarseUniverseSelection(self, coarse):
        self.Debug(f"Universe selection time: {self.Time}")               
        self.selectedSymbols = [cf.Symbol for cf in coarse of coarse.Price > 10]
        return selectedSymbols

Hello Fred and Alexandre, thanks for the replies 😊

Is your code working though Fred? I still get a key error in self.ActiveSecurities even when adding securities the way you do it. 

Fred - Maybe the solution is your way of creating a custom universe? I would love to see it. My idea is to select some stocks at midnight through coarse and possibly fine selection and at a given time before market open check the most current price in order to select the final watchlist. In my example, it would be to remove stocks from self.selectedSymbols and self.ActiveSecurities (through self.RemoveSecurities) inside the “CheckStocks” function. But for that to work, all stocks from universe selection must be in self.ActiveSecurities..

Alexandre - Using simple universe selection is not enough as I need to check the price at a given time right before market open (09:25 in this case), and possibly also other attributes than price. Maybe there is a better way of doing it than what I am trying to do?

Haakon Flaarønning

If we follow the recommendation of Alexander Catarino   

    def Initialize(self):
       self.UniverseSettings.ExtendedMarketHours = True

    def CoarseUniverseSelection(self, coarse):             
       self.selectedSymbols = [x.Symbol for x in coarse if x.Price > 10]
       return self.selectedSymbols

the results will be

2022-04-11 09:25:00 :            Check stocks time: 2022-04-11 09:25:00

2022-04-11 09:25:00 :            0 of 6674 not in active securities

2022-04-12 00:00:00 :            Resetting universe: 2022-04-11 21:00:00

2022-04-12 00:00:00 :            Universe selection time: 2022-04-12 00:00:00

2022-04-12 09:25:00 :            Check stocks time: 2022-04-12 09:25:00

2022-04-12 09:25:00 :            0 of 6658 not in active securities

2022-04-13 00:00:00 :            Resetting universe: 2022-04-12 21:00:00

2022-04-13 00:00:00 :            Universe selection time: 2022-04-13 00:00:00

2022-04-13 09:25:00 :            Check stocks time: 2022-04-13 09:25:00

2022-04-13 09:25:00 :            0 of 6674 not in active securities

Vladimir: Thanks for the reply. Yes, I know you can use a standard universe like that..the problem is though, as far as I know, you can't change that universe other than through the CoarseUniverseSeleciton function that runs at midnight. The purpose of my code is that I want to change the universe after checking the price in the try-catch statement in the CheckPrice function - when the time is 09:25 each day. 

Hi All,

Thanks for your replies. The OP asked for “[…] as I want to create my own “private” universe and remove and add securities when I please”.

Haakon, is that still something you want to do? You can do this through custom universes' “selector”… It seems you still want to do that but I want to make sure before I commit any more time to this.

Thanks,

Fred

Hi Fred,

Yes, I still want to create my own “private” universe. I have expanded on the code in the attached backtest. In the example, I exclude all stocks with a price less than 1 at 09:25 AM each trading day. It is important that this selection happens at 09:25 and not at midnight, as filtering on price is just an example. 

Please elaborate on custom universe selector :) from what I have read in the docs (https://www.quantconnect.com/docs/algorithm-reference/universes#Universes-Custom-Universe-Selection), all I can see is that it reads securities from a custom file..? I want to use coarse as it gives me all tradable securities on a given day. However, I need to control the universe selection at another time than midnight each day.

Hi Haakon

It is possible to create a dictionary with each iterated Symbol in Coarse Universe Selection as keys. Then update their values as the prices at 9:25.

def Initialize(self):
    self.UniverseSettings.ExtendedMarketHours = True

def CoarseUniverseSelection(self, coarse):
    selected = []    
    self.data = {}

    for c in coarse:
        if <pre-selection-criteria>:
            symbol = c.Symbol
            selected.append(symbol)

            if symbol not in self.data:
                self.data[c.Symbol] = None

    return selected

def CheckStocks(self):
    for sec in self.Securities:
        self.data[sec.Symbol] = self.Securities[sec].Price

Please note that you'll have to do a "pre-selection" in CoarseUniverseSelection first as if we include all symbols, there could be memory issues.

Best
Loui

Hi Haakon,

Short note: I DMed you.

F