I have securities that are removed from my universe that I'm trying to exit, but I'm unable to do so. I'm not sure if this is because of an issue with the data, the underlying security (delisted, bad volume, etc.), or my code.
When I want to exit a position, I cancel all existing open orders, then place the security in a list (self.close). This happens at midnight right after UniverseSelection in the OnSecuritiesChanged method. Another method runs through that that list at 10:00AM on market open days and places trades as required (sell or cover).
Here is the relevant code:
# is called whenever the universe changes
def OnSecuritiesChanged(self, changes):
#self.Debug("{} Securities Changed".format(self.Time))
self.open = []
self.close = []
for security in changes.RemovedSecurities:
self.CancelAllOrders(security.Symbol)
if security.Invested:
self.Debug("{} Removed Security {}".format(self.Time, security.Symbol))
self.close.append(security.Symbol)
else:
self.Debug("{} Removing security not invested in {}".format(self.Time, security.Symbol))
for security in changes.AddedSecurities:
self.CancelAllOrders(security.Symbol)
if not security.Invested:
self.Debug("{} Added Security {}".format(self.Time, security.Symbol))
self.open.append(security.Symbol)
else:
self.Error("{} Adding security already invested in {}".format(self.Time, security.Symbol))
def UniverseSelection(self, coarse):
#self.Debug("{} Universe Selection".format(self.Time))
# only use securites from US market with positive pricing
filtered = list(filter(lambda c: (c.Market == "usa") and (c.Price > 0) and (c.Volume > MIN_VOLUME), coarse)) # and (c.HasFundamentalData)
for f in filtered:
if f.Symbol not in self.universe:
history = self.History(f.Symbol, 25, Resolution.Daily)
self.universe[f.Symbol] = SecurityData(f.Symbol, history)
sd = self.universe[f.Symbol]
sd.update(f.EndTime, f.AdjustedPrice)
remaining = list(filter(lambda sd: sd.a, self.universe.values()))
remaining.sort(key = lambda sd: sd.b, reverse = True)
selected = [ sd.symbol for sd in remaining[:MAX_POSITIONS] ]
if len(selected) > MAX_POSITIONS:
self.Error("{} Selected more securities than required {} > {}".format(self.Time, len(selected), MAX_POSITIONS))
return selected
def CancelAllOrders(self, symbol):
self.Debug("{} Cancelling all orders for {}".format(self.Time, symbol))
openOrders = self.Transactions.CancelOpenOrders(symbol)
for oo in openOrders:
if not (oo.Status == OrderStatus.CancelPending):
r = oo.Cancel()
if not r.IsSuccess:
self.Error("{} Failed to cancel open order {} of {} for reason: {}, {}".format(self.Time, oo.Quantity, oo.Symbol, r.ErrorMessage, r.ErrorCode))
The securities in question are MGT and RLOC. Here are some logs:
2018-10-10 00:00:00 2018-10-10 00:00:00 Cancelling all orders for MGT TEMHQPA3FAZP
2018-10-10 00:00:00 2018-10-10 00:00:00 Removed Security MGT TEMHQPA3FAZP
2018-10-10 00:00:00 2018-10-10 00:00:00 Cancelling all orders for RLOC UMQPTIBCU8V9
2018-10-10 00:00:00 2018-10-10 00:00:00 Removed Security RLOC UMQPTIBCU8V9
2018-10-11 00:00:00 2018-10-11 00:00:00 Cancelling all orders for MGT TEMHQPA3FAZP
2018-10-11 00:00:00 2018-10-11 00:00:00 Removed Security MGT TEMHQPA3FAZP
2018-10-11 00:00:00 2018-10-11 00:00:00 Cancelling all orders for RLOC UMQPTIBCU8V9
2018-10-11 00:00:00 2018-10-11 00:00:00 Removed Security RLOC UMQPTIBCU8V9
2018-10-11 00:00:00 2018-10-11 00:00:00 Cancelling all orders for NXTDW VTTIWRG7CIN9
2018-10-11 00:00:00 2018-10-11 00:00:00 Added Security NXTDW VTTIWRG7CIN9
2018-10-19 00:00:00 2018-10-19 00:00:00 Cancelling all orders for MGT TEMHQPA3FAZP
2018-10-19 00:00:00 2018-10-19 00:00:00 Removed Security MGT TEMHQPA3FAZP
2018-10-19 00:00:00 2018-10-19 00:00:00 Cancelling all orders for RLOC UMQPTIBCU8V9
2018-10-19 00:00:00 2018-10-19 00:00:00 Removed Security RLOC UMQPTIBCU8V9
2018-10-19 00:00:00 2018-10-19 00:00:00 Cancelling all orders for WHLRW VQ5L08FL3F51
2018-10-19 00:00:00 2018-10-19 00:00:00 Added Security WHLRW VQ5L08FL3F51
2018-10-25 00:00:00 2018-10-25 00:00:00 Cancelling all orders for ENRJ VRGUGPZORMUD
2018-10-25 00:00:00 2018-10-25 00:00:00 Removed Security ENRJ VRGUGPZORMUD
2018-10-25 00:00:00 2018-10-25 00:00:00 Cancelling all orders for MGT TEMHQPA3FAZP
2018-10-25 00:00:00 2018-10-25 00:00:00 Removed Security MGT TEMHQPA3FAZP
2018-10-25 00:00:00 2018-10-25 00:00:00 Cancelling all orders for RLOC UMQPTIBCU8V9
As you can see, these securities are never removed from my portfolio. It is also specific to these securities in a backtest run from 2015-2020. For other securities, my code exits positions properly.
I believe this is a security data problem because the documentation on Universes for SecurityChanges says:
# Gets the symbols that were removed by universe selection. This list may
# include symbols that were removed, but are still receiving data due to
# existing holdings or open orders (IReadOnlyList)
self.RemovedSecurities;
This leads me to believe that I'm unable to exit because of volume. I'm wondering how I should handle cases like this.
Also unrelated:
Does the self.UniverseSettings.MinimumTimeInUniverse property utilize the self.UniverseSettings.Resolution property or is it fixed to days? For example:
self.UniverseSettings.Resolution = Resolution.Minute
self.UniverseSettings.MinimumTimeInUniverse = 1000
Will this require securities to be in my universe for 1000 minutes or 1000 days?
AK M
I wanted to provide an update. I believe the issue was the securities becoming illiquid over the duration I was holding them. I was unable to act on this because of an error in my UniverseSelection function.
In my code, I filter the list of CoarseFundamental options, update the filtered results, and select the symbols that match my criteria. The issue is that I only update the securites that are filtered and never update the securites already in my universe. So sometimes, securities that initially met my liquidity requirements stopped meeting them, and thus were never updated again.
Here is my updated code:
def UniverseSelection(self, coarse):
#self.Debug("{} Universe Selection".format(self.Time))
# save reference in dict
current = {}
for cf in list(coarse):
current[cf.Symbol] = cf
filtered = list(filter(lambda c: (c.Market == "usa") and (c.Price > 0) and (c.Volume > MIN_VOLUME) and (c.DollarVolume > MIN_DOLLAR_VOLUME), coarse))
for f in filtered:
# add new symbol to universe
if f.Symbol not in self.universe:
history = self.History(f.Symbol, 25, Resolution.Daily)
self.universe[f.Symbol] = SecurityData(f.Symbol, history)
# update sd and remove any securities no longer in universe
new = {}
for symb in self.universe:
sd = self.universe[symb]
if symb in current:
sd.update(current[symb].EndTime, current[symb].AdjustedPrice, current[symb].DollarVolume)
new[symb] = sd
self.universe = new
remaining = list(filter(lambda sd: sd.a, self.universe.values()))
remaining.sort(key = lambda sd: sd.b, reverse = True)
selected = [ sd.symbol for sd in remaining ]
return selected
Â
Derek Melchin
Hi AK,
The reason MGT and RLOC are never removed from the universe is because LEAN can't exit the positions. There is no volume for these securities since they were both delisted (see the related related PRs: MGT and RLOC). RLOC moved to an OTC exchange, and we don't have OTC stock data.
Universe selection is built to deal with delistings automatically, but as the attached backtest shows, LEAN doesn't recognize MGT's delisting. This is related to a data issue for MGT and RLOC. The issues have been reported here.
When it comes to the MinimumTimeInUniverse property of UniverseSettings, assigning an integer value translates to a day count, regardless of the universe resolution. That is, writing
self.UniverseSettings.MinimumTimeInUniverse = 1000
requires securities to stay in the universe for a minimum of 1000 trading days. Instead, we can assign a timedelta object to universe settings if we want to state other time periods. For instance, we can use
self.UniverseSettings.MinimumTimeInUniverse = timedelta(minutes=30)
to demand securities spend a minimum of 30 minutes in the universe. But this duration is effectively rounded up to the closest number of full days as the universe is refreshed on a daily basis.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
AK M
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!