Share New Research 
Let's assume it's a simple file that contains a list of 0-20 symbols I *may* want to act on during the early hours of the next trading day.
I want a QC algorithm that will enable me to backtest my symbol list, as well as eventually live trade with it.
For backtesting, I want the QC algorithm to load up a file that contains a date and a symbol, and there may be more than one symbol per date. I want the algorithm to be able to load up the enough historical data on these symbols to warm up a SMA or other such indicator that requires prior data. And, I want to be able to process the bars for each of these symbols with the appropriately warmed up indicators to apply other rules that I want to code to potentially act, or not act on these symbols.
I then want to be able to evaluate the overall performance over a period of time of the various symbols that were bought and sold.
When satisfied, I want to move the program into live mode to be able to automatically buy/sell these securities through my Interactive Brokers account.
What I intially tried was creating a derived class from BaseData and loading up the data I wanted via the GetSource() and Reader() overrides. While I could get the data loaded up, and in spite of calling AddData() in the Intialize() method, I wasn't getting called on any bars for these symbols in the OnData() callback.
I think the BaseData approach I mentioned above is more for using other historical data to drive the backtesting rather than just loading up a list of symbols to potentially promise. After more hunting around, I found the "DropboxUniversalSelectionAlgorithm" example. I think this is closer to what I want. I believe this algorithm should simply pre-load the list of symbols I may want to process for the day and get them added to the list of securities to feed to the OnData() callback. Is that correct?
What's still unclear to me is how to "warm up" indicators such as a SMA or EMA for a new symbol that may appear on Day X but I really need enough historical data going back Day X-20 to fill these indicators.
Am I starting to get on the right track? I'm a huge Learn-By-Example guy - any other great algorithms anyone is willing to share that do something similiar that I can try to clone and work from?
And, is there a way to "in hindsight" warm up some indicators with historical data to help use things like SMA in OnData()?
Thanks in advance for any help.
Jared Broad
Hi Ted, the problem you're describing has been addressed many ways; I'll give you a few of the tools and you can assemble them how you like:
1 - Scheduled Events - Run code snippet at prefixed times each day.
2 - Basic C# WebClient - Pull string from external source and parse it how you like.
3 - User Defined Universes - Provide a list of symbol strings & dates in the formats we consume and the universe selection logic in LEAN will automatically add those symbols to the universe for you. This is the most "LEAN" way of doing it.
4 - AddSecurity / RemoveSecurtity methods - You can dynamically add and remove securities as you like.
5 - History / WarmUp - History() API is the best for your purposes - get historical data from a stock you're subscribed to. Pass that into whatever methods you like (including indicator warm up).
> I'd recommend using Scheduled Events to load your file, parse it, then feed it into a user defined universe.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
TedVZ
Great, thanks again for the prompt response. You guys ever sleep? :)
I'm sure these problems have mostly been solved, it's just a matter of me trying to find the best patterns to follow, so thank you very much for the pointers.
My biggest concerns is the adding of a new symbol but also trying to get historical data on the same time. Basically, if in the early morning hours of 11/18/2016 I import a stock/signal, I go to process the signal after the market opens and i try to execise some logic in OnData such as "buy iff signal is > smaFast && smaFast > smaSlow". So long as I can use History() or some mechanism to add a symbol on Day 0 and be able to use a SMA or other indicator that VERY SAME day even though it may require historical data.
I'll look into the various pointers you provided. Thanks very much.
Jared Broad
Yes I think it accomodates your needs; the history API is running on the same live stream as the market data and requests can be filled up to "a seconds ago"-ish. This will allow you to warm indicators on the same day.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Alexandre Catarino
In LEAN github, there is an example of the LEAN way of doing it:
Dropbox Universe Selection Algorithm.
I wrote an alternative solution following Jared's recommendation:
Schedule Events - AddSecurity - RemoveSecurity combo.
Everyday at 9:20, an event is fired to update the universe. WebClient is used to read a remote Dropbox file. If there is a new security, adds it to the user universe. If a security was removed, remove it from the user universe. Fetch data from the History Server for all symbols listed in the remote file for that particular date.
Note that when we remove a security, it is still part of the IAlgorithm.Securities object, because this object holds information about past trading performance, but it won't be part of the TradeBars object that arrives at the OnData event handler.
Since you want to use indicators, the solution we usually recomment is using a "SymbolData" class to cache the indicators (here is an example: EMA Cross Universe Selection Algorithm)
TedVZ
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!