Is there any option I missed or any plan to implement peristent storage for own preprocessed/generated data in research environemnt (it would be great if it can be accesed from alogs too)?
In research phase there is quite common pattern, that you will preprocess data first (e.g. generate custom higher frequency data from minute data, or some ML features..) and then work with them many times.
This preprocessing often takes long time (tens of minutes, hours). So in local environment it's good idea to serialize them (e.g. to csv, parquet, database) and work with them repeatedly.
I didn't see anything like this in Quantopian.
It's quite waste of (computational and human) resources when you generate same data many times, so I believe it woudl be of great use for many users.
Something like encapsulated S3 like storage for pandas dataframes would be ideal (and cheap enough for reasonable amount of data, relatively to price of computational resources..)
Oldrich S
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!