Share New Research 
Hello there,
I am trying to develop a custom Risk Management decision tree so that when a condition is met, I close the position and remove the underlying from the portfolio (otherwise, OnData Loops back on it and I want to avoid this).
However as you can see in the attached backtesting, the securities are not removed, even when I loop through each component of the portfolio to force removing everything - underlying equities remain there in the portfolio... Have you guys ever witnessed this behavior? Is there another way to do this ? It seems to be working when I'm resetting the portfolio every night so I tend to think that there is a problem with intra-day behavior...
Thanks in advance for your help,
Terence.
Alethea Lin
Hi Terence,
The RemoveSecurities() command will not take a security out of the Portfolio, and that’s why you will still see the security in your Debug message. However, it will liquidate the positions and cancel any open order, and the security will be removed from the data feed and marked as nontradable until the next Universe Selection.
In addition, if the universe is not too big, we recommend using OnSecuritiesChanged() to keep track of a list of symbols and look through those symbols instead of removing securities. If the universe is too big, you might run into issues regarding subscription limits with the brokerage, in which case you would need to remove securities from the data feed. Check out this documentation to learn more!
Hope this addresses your question. Thanks for your support!
Terence Mahier
Hello Alethea,
Thank you very much for your answer but my issue is that RemoveSecurities() does not seem to remove the ticker from the data feed since the algorithm is reinvesting right away after the position was closed by risk management.
Is it possible that the function stockDataSource() is only sensitive to the date and not the time, therefore adding the stock back into the data feed ?
On the other hand, I cannot use OnSecuritiesChanged() for the reasons explained in the post below.
https://www.quantconnect.com/forum/discussion/6015/option-trading-on-custom-data-onsecuritieschanged-behaviour-problems/p1Alexandre Catarino
Hi Terence,
I don't want to run away from the issue because adding and removing securities manually in a universe selection algorithm is fine, but it breaks the universe selection pattern. If the algorithm is not adding hundreds of securities, using RemoveSecurity is not the best alternative, especially combined with AddEquity.
I would suggest a different approach: create a list with the underlying symbols which options the algorithm is allowed to trade. Instead of removing the security with RemoveSecurity, remove the symbol from the list and, consequently, the algorithm will not loop through data.Keys but the created list.
After these changes, we may take a look into the issue of not being able to use OnSecuritiesChanged, This method is part of the universe selection pattern and I think this algorithm would profit from its addition.
Alexander Ermushev
Am I correct understand, when iterating over changes.RemovedSecurities in OnSecuritiesChanged method, you don't need to use algorithm.RemoveSecurity? It is necessary to remove data only from custom lists and dictionaries.
Jared Broad
Correct; RemovedSecurities have already been removed from the algorithm and this is a notice allowing you to clean up any lists/data in your algorithm that LEAN does not manage.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Cesare Augustus
Hi, in Alethea's quote (below), what was meant by "big"? How big is "big"?
I quickly checked online and it appears that with Interactive Brokers, "by default users can obtain the real time market data of up to 100 instruments simultaneously." -- Is this the case? Would I get an error if I went live and tried to subscribe to more than 100 securities wit IB? If so, I may need to rethink my algo (in development), because currently it's subscribing to about 3,500 securities at the midnight Universe Selection, as it's only filtering by Price and Shares Outstanding.
Alethea's quote: "In addition, if the universe is not too big, we recommend using OnSecuritiesChanged() to keep track of a list of symbols and look through those symbols instead of removing securities. If the universe is too big, you might run into issues regarding subscription limits with the brokerage, in which case you would need to remove securities from the data feed."
Jared Broad
Alethea's quote applies to using the IB data feed. QC's data feed does not have the same restrictions.
There is no good way to do mass-market intraday analysis in QC at the moment. We're aware of the demand for a feature like this and are thinking of the best way to make it happen. It will likely be a DB-query service than a full feed.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Shaun T
Please help, I really want to remove a security from the data feed intra-day but don't see any way to do this. Reading over various forums posts, I'm still not sure if its possible or not. Is universe selection the only method of removing stocks from the data feed, I assume that is what is meant by "Alethea Lin" above when she says: "If the universe is too big, you might run into issues regarding subscription limits with the brokerage, in which case you would need to remove securities from the data feed. Check out this documentation to learn more!"
I also see a post here:
https://www.quantconnect.com/forum/discussion/1491/load-list-of-symbols-to-process-each-day/p1Which says "Note that when we remove a security, it is still part of the IAlgorithm.Securities object, because this object holds information about past trading performance, but it won't be part of the TradeBars object that arrives at the OnData event handler."
I don't see this behaviour when I run.
Here is my issue, I want to do similar to what Jared says above "mass-market intraday analysis in QC", but I don't need (and perhaps might not even fulfill my need) some DB-Query service, I just want to create a scheduled event to run 10-20min before the market open which loops over a large list (hard coded, like 2000 or so) of equities and calls AddEquity, make a decision to keep or not based on results from a couple History calls, then RemoveSecurity if I don't like what I see. I did this in a loop and its very fast, I could in theory process all 2000+ equities in under a few minutes but the issue is that RemoveSecurity never removes the security from the data feed and the program can't effectively run after all these AddEquity calls. I mainly want to filter on premarket volume and gap up/down from previous day. It seems simple enough but QC doesn't properly remove a security from the data feed when RemoveSecurity is called from what I see.
Universe selection doesn't work for what I need I believe, unless I could get it to do universe selection multiple times 10-20 minutes before the open where I would add say 300 stocks at a time to the universe and then on the next universe call remove those and add another 300, repeat, until I curate my final watchlist, then make one more universe selection call, can this be done? (BTW this assume universe selection actually removes the securities from the data feed even when called multiple times a day?) I thought about this but don't see a way to call it multiple times before the market open.
Jared Broad
Mind posting an example to support to demonstrate "RemoveSecurity not working". RemoveSecurity removes it from the user-defined universe (i.e. when manually added via AddEquity) - if you added it from coarse you'll need to remove it via coarse.
All regression and unit tests are passing so we'll need a properly reported example to investigate further.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Shaun T
Thanks, I've attached an example of what I'm seeing here.
Derek Melchin
Hi Shaun,
To be able to remove security subscriptions intraday, we need to reduce the MinimumTimeInUniverse.
UniverseSettings.MinimumTimeInUniverse = new TimeSpan(0, 1, 0); // 1 minuteWe can then schedule the universe scanner to iteratively run prior to the market open by using
for (int i = -20; i < 0; i += 5) { Schedule.On(DateRules.Every(DayOfWeek.Tuesday), TimeRules.AfterMarketOpen(EquitySymbols[0], i), FindStocksToTrade); }When the scanner is running, we should only call AddEquity on securities that meet the universe criteria.
public void FindStocksToTrade() { ... foreach (var fsip in FSIPSymbols) { var symbol = QuantConnect.Symbol.Create(fsip, SecurityType.Equity, Market.USA); var histBars = History<TradeBar>(symbol, 60, Resolution.Minute); foreach (var histBar in histBars) { // Do stuff } if (add) // If we want to add the symbol { AddEquity(fsip, Resolution.Minute, Market.USA, fillDataForward, Security.NullLeverage, extendedMarketHours); } else { RemoveSecurity(symbol); } } }See the attached backtest for reference.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Shaun T
Thanks for the info. I was able to create a function to do exactly what I wanted. I just call it once about 20min prior to the market open, create the symbol and call history for the daily (single day, to capture yesterday's close) and then history for minute data for the premarket, then remove security if I don't want to keep it or add equity if I do. Testing showed that the second history call behaved a bit unusual (looks like a bug). When requesting the minute history it would go all the way back to the close of the prior day, doesn't matter what value I give it (although I believe a value of 1 was different, but anything greater than one had the effect of providing data to yesterday rather than the actual value I provided). This doesn't bother me though, I filtered out the older bars. I also found that if I ran this function when the market was open, requesting enough bars to step into the premarket by any amount would trigger the full history of the premarket to be returned (and probably to yesterday again). I don't remember all the specifics, as I just worked around them, and it was a number of days ago now. Just thought I'd let you know. Another issue I had was the messages to the log about symbols not having data. When I want to backtest my algorithm the 1000+ stocks I check are based on what exists today, so I got a lot of messages during the backtest about data not available, it ran okay but filled up my log space after processing about a month of data. I realized I can avoid this by checking the symbol.ID.Date value to skip symbols that haven't been added to the market yet, so all is good now.
Terence Mahier
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!