Appending symbols to Coasre Universe

Debuging this a bit more… My invested stocks are in the first positions in my stock list that is returned from the Coarse function.  So it looks like they are returned, but I never see them when I loop through symbols in onData.  Why would this be the case?

Hi John J,
We can check the securities added or removed after universe selection by implementing the OnSecuritiesChanged event handler. Would you mind attaching a backtest highlighting the issue for a precise diagnosis?
Best,

Varad Kabade

Thanks Varad,  I have attached my backtest.  I've added a lot of debugging code to try and figure whats going on.  It sure seems like I'm appending my invested symbols to the list returned by the coarse function but they never make it to onData.

I tried this coarse function as well and its not returning anything.  It's using a very similiar list comprehension (compared to ‘stocks’ variable).  

On the first pass ‘own’ is 0 and ‘stocks’ is returned.  Inthe onData section a few purchases are made then by the time it loops through a couple more times it registers ‘own’ is no longer empty and returns ‘own’  ‘own’  is a list of Symbols just like ‘stocks’.  How come this list of symbols never gets looped through in onData? 

      def CoarseSelectionFunction(self, coarse): 
        invested = [ x.Symbol for x in self.Portfolio.Values if x.Invested ]
      
        sortedByDollarVolume = sorted(coarse, key=lambda c: c.DollarVolume, reverse=True)
        own = [x.Symbol for x in sortedByDollarVolume if x.Symbol in invested]
        self.Debug(f"own length is {len(own)}")
        stocks = [x.Symbol for x in sortedByDollarVolume if x.Price > 15 and x.DollarVolume >= 900000 and x.HasFundamentalData == True] #HasFundamentalData effectivley removes indexs/ETFs

        if len(own) > 0: return own
        return stocks[:5]

Any further suggestions on what to try?

Hi John,

Looked at your code. It's not trivial…

I rewrote a bit so I own so to speak the debug logs. I see errors in your orders because you are missing cash (margin). Except from that, I don't see problems. I see your invested tickers returned by Coarse. And I see all Coarse being in OnData.

So, what I see is incapacity to buy and then that points towards your condition for sell never being true. And indeed, in the logs, you don't sell.

I did not debug enough to understand why exactly you don't sell however. This is left as an exercise. 😊 Backtest included.

Fred

John J. 
Here's my attempt to rebuild your algorithm a bit to avoid insufficient buying power. 
In my opinion, it works as you planned. 

Thanks for taking a look at this, but I'm still having trouble…

My overall goal is that I'm looking for an indicator condition to leave my invested positions, however I need to be sure that my invested stocks show up in onData so the condition can be checked.  The solutions I see are:

• include every stock into my universe (1000s!)
• append my invested symbols into my universe

Option 2 sounds easy, but I'm simply not seeing these stocks in the onData section.  Unless I'm missing something…

I added a backtest to illustrate the issue.  I'm expecting data from the securities I'm invested in to be looped though in self.Data.keys.  However when looking for them I'm finding they are not there.  

The number of symbols in onData should equal the number returned from coarse.  If you look at my two debug lines they clearly are not.  What gives?

John J
Could you please illustrate the problem in our version of the code.   

Take a look at the last backtest I attached and let me know if you need something else.  That one is based off of your code.  I just added a couple debug lines and lists to show the issue.  

I removed the selling portion as well to illustrate the issue more.  I.E. If I own AMZ and return AMZ from coarse then it should always be looped through onData… 

I'm finding the basis of the problem still exists: symbols returned from coarse ≠ symbols in onData

John J,

Why did you comment out the selling part of the code? 

        #  for sec in self.Portfolio.Keys: 

        #     if sec not in self.selected:

        #         self.SetHoldings(sec, 0)

I removed the code that sells the securities to help illustrate the issue.  As the algorithm buys more securities then they should be appended back in coarse and returned back to onData. (but I'm finding they are not)

I changed the holding percentage to 0.01 so I could buy a good amount of stocks before running into a buying power issue..  Ultimately I would be checking a indicator condition (i.e. EMA crossover) to initiate a sell.  However since the stocks returned in coarse ≠ the stocks found in onData, then as I loop through the stocks in onData I never actually find the stocks I own!  even though these were added and returned in coarse.  Its important all the stocks from coarse are passed to onData, otherwise I won't be checking for a sell condition and miss my exit.

I don't understand why I return 8 stocks from coarse but then I only find 5 stocks in onData.    Shouldn't the length of the symbols returned from coarse equal the length of symbols found in onData?

Hi John,

When you look for symbols in OnData:

for symbol in self.Data.keys():
           symbol_values_in_onData.append(symbol.Value)
           symbols_in_onData.append(symbol)

it looks in symbols that are in your Data dictionary, and not in the data dictionary passed to OnData (parameter “data”). The symbols passed to OnData are in there.

Check that attached backtest please.

Cheers,

Fred

Thanks Fred, I think that was it.