Share New Research 
I have noticed that the price values of stocks in a coarse universe on occasion seems to be way off the correct value for a stock a given date. I have checked both the raw and adjusted price, an although they differ, they are still way off. I have attached a backtest which calculates how much each stock has gapped up since previous day close, however, the previous day close is simply wrong. The logs display (for date: 2021-05-11):
{'ticker': 'TANH', 'gap': 21971.42857142857, 'symbol': , 'last_close': 1.12, 'cur_price': 247.2}
{'ticker': 'TBLT', 'gap': 14402.164502164504, 'symbol': , 'last_close': 0.693, 'cur_price': 100.5}
The “last_close" value comes from coarse universe selection and are way off. Neither the CoarseFundamental.Price, CoarseFundamental.AdjustedPrice or CoarseFundamental.Value gives the actual previous day close.
How come these values are way off? Do I need to perform a history request at the end of each day to get the actual closing price that day?
Haakon
Extra note: The minute history request is not too great as it only fetches data for about 1000 of >9000 stocks. With fillForward=True I would expect a result even if the stock does not have data for the minute in question. Are history requests unreliable? Is it better to subscribe to all securities and wait for data to come in at a specific time (09:25 in this case)? I get about 4000/9000 stocks by doing so, so not great that either..
Alexandre Catarino
Hi Haakon Flaarønning
How do you know prices are way off? Are you comparing them against another source? Which one?
The historical data requests for daily data should give you the same value.
Best regards,
Alex
Yuri Lopukhov
Haakon Flaarønning I believe there is a data issue for these two symbols, so its really best to make a data issue. Not really sure where it is done now and why they made it so damn difficult though.
As for minute history data, note that if you request 1 minute data for only 1 minute, esp. in pre-market, you will only get those securities that traded this specific minute. So that is ~1k securities I suppose, which had ticks within this minute. I am pretty sure that FillForward parameter will only work within the selection, so it will fill the gaps to the right, as long as your selection includes some data on the left. So you may want to extend time period to get more stocks and only pick last minute for each. For example, request data since 9:00am to 9:25am.
Haakon
Alexandre Catarino, Yuri Lopukhov Thanks for the replies! 😊
Alexandre Catarino I have compared to multiple sources (Yahoo, TradingView)..but you can also see that the daily price fetched from coarse selection is way off compared to the minute history request depicted in the “cur_price” key in the log above. Surprisingly, a history request for that symbol that day (TANH, 2021-05-11) gives the correct closing price - something is off with the price obtained from the coarse object.
Yuri Lopukhov There probably is a data issue for the coarse object, however, it is not only a problem for these symbols. I haven't checked how big a problem it really is, but through calculating the gap of all tickers over a period of time I can see many symbols getting a a gap >4000% clearly indicating false data. Thanks for the idea of extending the minute history request. Fetching 1 minute data for 15 minutes (09:10 - 09:25) gives approx 3000/9000 symbols which atleast is a bit better. Wondering if the symbols it doesn't manage to fetch data for has really low trade volume during pre-market?
Derek Melchin
Hi Haakon,
If we match the data normalization mode (raw/adjusted) of the CoarseFundamental object and the history request, we get prices that are relatively close. See the attached backtest for reference.
Best,
Derek Melchin
Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Haakon
Derek Melchin Thanks for the reply!
Surely the prices was edited recently? Otherwise I am not sure what happened, because I was sure I checked the AdjustedPrice as well.
In any case, I reverted to use history requests to fetch last day close. It takes a while to run in the algo, but it ain't too bad.
By the way, this was an example of a simple scanning feature to identify top gappers at a specific time during the day. I know that such scanners are very computationally heavy, and even more so when everyone has to implement it themselves in each algo. Are there any best practices for implementing such scanners? I am thinking forexample history requests vs subscribing to equities. Is it something you are working on implementing as a feature in QC? Scanning for attributes such as gap, relative volume etc at certain times of the day can be very useful.
Derek Melchin
Hi Haakon,
It's possible. We reprocess the coarse files daily to update the adjusted price and it can also include patches for past raw values.
We don't currently have this feature built-in. Like you mentioned above, it's only currently possible with Coarse + Scheduled universes or with Coarse universe + Scheduled Events + AddEquity/RemoveSecurity. The technique of history requests vs subscribing to all the equities depends on the size of the universe, the data resolution, and the selection frequency. If we subscribe to all the listed US Equities instead of using history requests, the algorithm will take a long time to run. Consider running some tests and sharing the results with the community.
Best,
Derek Melchin
Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Derek Melchin
Hi Yuri,
Please see Report New Issues.
Best,
Derek Melchin
Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Yuri Lopukhov
Hi, Derek Melchin
Documentation states this:
Just like that, I am supposed to see this page in my algo lab? Where is it?
I found this link if I open a dataset and scroll its about page:
Why make it so complicated and well hidden? This doesn't make reporting data issues easy at all. It was easier before.
Derek Melchin
Hi Yuri,
There is no link from the Algorithm Lab to the Data Explorer Issues page, but we need to log in to create data issues. To access the Data Explorer Issues page, we can click the link in the docs.
The UX team is working on upgrading the Data Explorer to make the process easier.
Best,
Derek Melchin
Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Haakon
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!