Share New Research 
I'm currently working on an approach that involves massive data sets. The way the current algo approaches things involves search and performing data comparisons (moving averages, bollingers, standard deviations and several others) on 1 years data on some 1500 + equities. This is done before market open using the CoarseUniverseSelection process. Data objects are created to store these indicators for all 1500 symbols. Then in "On Data" we examine the incoming prices on this universe by the minute and compare that to our stored object data to make buy sell trading decisions intra-day.
So we are essentially fetching 1500 stocs x 252 days of data, before market open. In the first minute of the ay we are then reducing that universe down to a select number (e.g. 30) securities for which we want to trade.
My question is not whether the QC - AWS system can handle the task efficiently. It can and does. The whole thing takes less than a few seconds to run. My question is when we go live . Will we be having to bpay large scanning costs because we have surpassed certain data feed limitations or will we be subjected to data throttling by IB?
If the latter is the case, we will have to re-architect our whole approach: Hopefully someone here can allay my fears and tell me I need not be concerned.
Thanks
Serge d'Adesky
Forgot to add the link to IB'S policies on this: ib's pacing policies
Michael Manus
Serge the link you pointed to is about historical data limitations.
Downloading data for 1500 stocks with IB would take months depending how much you need.
What i have heard the historical data requests are coming from QC directly (data from their provider)
Livetrading is somehing different.
I am not a live trader but when going live you could choose between IB data and QC data.
IB limits to 50 subscriptions minimum. Executing orders does increase it every month. to i dont know. i have seen 500 subscribed stocks at a time.
you could trade 30 (31 with spy for benchmark) and be safe,
Dear Jared Broad it would be cool to point him to the GOING LIVE section in de DOCUMENTATION because his question comes up every month and you post is not up to date. I dont know for sure if he could set live data feed from QC. (i have read it here somewhere in the community).
Serge would be safe with 31 with IB subscription data feed.
Last question about data i answered maybe incorrect:
Your docu for going live:
Jared Broad
If you use QuantConnect as a data source it pulls from our internal systems which are built from ticks. Most users hitting massive history requests run into the limitation that we restrict algorithms to 10 minutes per time loop. The next step is for them to break up the history request into smaller chunks.
As shown in picture above we also allow people to use IB as their data source but you are subject to all the normal IB restrictions. Eventually ending in you slowing down requests / doing them in smaller chunks as well. You must use IB for data if you're doing options or futures live trading.
Ultimately as fast as we try and make LEAN and the History service -- people are always finding its limits =). We've had a few questions in this area recently as people deploy more universe algorithms so it might be worth another look in the month ahead to see if we can make it closer to backtesting speeds.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Serge d'Adesky
Jared, Is there something I don't understand here? Is it possible to use QC as a datasource yet still place trades live with IB?
I will be calling IB to understand a little better how their billing works on this issue.
Thanks
Jared Broad
"Is it possible to use QC as a datasource yet still place trades live with IB" -- Correct (last time I checked). We automatically configure IBGateway to enable what IB calls "Blind Trading" (trading without having IB data).
The live tick stream is easy -- the hard part of your discussion-post is the 1500 history queries on a tiny VM, which was the reason for majority of my comment above. When you elect to use IB as your algorithm's data source we use IB's History API.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Serge d'Adesky
OK . I just talked to IB's api department. If I understood correctly, Sending thousands of 1 year history requests would certainly trigger IB's thottling mechanism, which would essentially put a bandwidth limitation on the process. You would not be charged more for the task, but you would need several hours of time to complete the process.
Once the universe is built, and in my case, the data objects storing the pertinent averages and indicators, there would not be any throttling issues with getting the ticks for those 1000 symbols at the first minutes of trading, since that would only need one data point, the opening tick.
So now my only question is : can I control the time the CoarseUniverseSelection is run, setting it to say 1:00 am so it is easily done by market open the next day?
I presume I would use the self.Schedule functionality for that?
Serge d'Adesky
Jared,
If we do use the QC history api but send the requests "blindly" to IB, is there any special coding needed in the algorithm to know how to do that? Or is that automatic when you select to go live, choose IB as the brokerage, then select QC as the datasource?
What would be a reasonable VM machine to select to be able to handle this amount of requests (1500 stocks, x 252 days ) without spreading them out over several hours. For example, a 10 minute processing time is wholly accepable.
Jared Broad
- With the current technology I'd batch them up to say 15 requests x 100 stocks // over say 15 min. The history fetching would probably be something you schedule for 6.00-6.15am.
- You cannot schedule the universe selection. It happens daily at roughly 4-6am when the data is ready.
- No special coding is needed to specify data sources.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Serge d'Adesky
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!