I encountered several wrong values in the historical data, especially when using Minute Resolution.
The values are so frequent and distant from reality that they mess up my backtests.
In order to have more realistic outputs from my backtests I would like to have a way to detect these outliers.
Here is an example of minute data consolidated into 15 minutes bars:
The same data in trading view looks like this:
I would assume the wrong data points are usually (but not always) a High or Low, by the higher chance for those to be wrong being watermarks.
As for now I had to change some of my strategies to somehow avoid using High and Low values, and rather create “my own” consolidated bar High and Low using the minute's min and max Close values.
I was thinking of other solutions including looking at the latest average variance/candle sizes and mark a trade bar as “suspect” if it is outside two standard deviations from it, and deny opening positions whenever we have “suspect” bars as part of the data for our decision. But this of course is too aggressive and makes me miss good opportunities.
Has anybody found a solution or is willing to share their current approach or ideas to solve the problem?
..and, has anybody experience this problem also live?
I will let the thread open for ideas and suggestions :)
Varad Kabade
Hi R G Quant,
The difference in data can be due to different normalization modes. The default in Lean/QC is Adjusted price data. There may be some because data was consolidated differently because of different timestamps. We advise looking into individual data points. We've created a portal for the submission and management of Data issues. This way the community can transparently identify any issues and have visibility when they are fixed. Can you please post your ticket here. See a quick video demonstration of how to use this tool.
Best,
Varad Kabade
R G Quant
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!