Share New Research 
I've been digesting what I can find to figure out how to convert some tick data I have to Lean data format. I've run across a couple of gaps in info that I would appreciate some clarification.
Looking at this doc
And some of the example futures data that is provided along with the market-hours-database.json file.
In the readme.md file in futures data folder there is no mention of timezone. Looking at the market-hours-database.json file does appear to specify “dataTimeZone” as UTC. Just wanting to confirm that.
In the docs linked above, the exchangeTimeZone for Comex and Nymex indicate Eastern timezone, however in the market-hours-database.json file, Comex is configured as Central timezone which I believe may be incorrect.
I was also looking through the tick trade data for the GC market that is provided in the future/comex/tick/gc folder in sample data and I see some ticks in that file that show a quantity of zero. That doesn't really agree with my definition of a trade tick and would like to know if this is invalid.
I appreciate the help here.
Randy Terbush
Moderator didn't like when I attempted to bump this to Jared Broad last week. Seems waiting a week is a reasonable amount of time before trying again to get some answers to basic questions here.
Randy Terbush
I'd like to also add another question here:
The example futures tick data for GC market shows an exchange column value of ‘usa’, even though it is placed in comex folder. Is this correct? Exchange in all futures data (at least for COMEX “market” in this case) is ‘usa’?
Fred Painchaud
Hi Randy,
I don't think I can answer all of your questions. But here is what I can contributed.
Normally, data is UTC, yes. I never checked if this is properly enforced across the board or not. But since normally data in itself is UTC, that makes the basis to shift to other timezones in the algo code…
Re the GC tick data in future/comex/tick/gc, are you referring to the data downloaded automatically with LEAN CLI when you start a project? If so, please note that to me, that 200MB of data is for the simple purpose of making algos somewhat run on something. I really don't think that data is to be used as a reference for anything. I'm not even sure it is always fetched from “the” source, the one that can be fixed over time through data issues tickets or by the devs themselves.
If you want to use your own tick data in LEAN, please refer to the custom data guide here: https://www.quantconnect.com/docs/algorithm-reference/importing-custom-data.
I never played myself with importing futures data (as this looks like what you want to do) but I do import equity data (at all resolutions except tick) without a problem.
Fred
Randy Terbush
Thanks for the response Fred Painchaud
That link you provide is dead. I think this might be the intended link:
That link doesn't really get into how to format data for the local data store from what I can see.
I am only using that data provided by a lean init as an example of how I need to format my own data. That, along with the info in the future/readme.md file. If that data and readme are not a proper reference for building this data store, I would love to get a pointer to the definitive guide on this.
I've successfully built a local data store for equities and am just trying to do the same for my futures data. I do think it is also important that we resolve the market times data as I believe the Comex market times are incorrect.
I'd also like to confirm with someone that the value of ‘usa’ is correct in all futures tick data for the “Exchange” column and that this should not map back to the actual market folder value.
Fred Painchaud
Hi Randy,
Well ok then. The link I provided got corrupted by the period I added at the end to finish my sentence. So yes, your link is good.
Sorry it does not help you however. I'm reading in a lot of equity data from multiple different folders on my computer and from multiple different formats (CSV, binary, plain text, etc) all fine. So in a nutshell, the custom data interface was designed so you do not have to know how LEAN data is formatted. You implement the format in a custom data class…
Cheers,
Fred
Randy Terbush
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!