Share New Research 
Greetings,
I've been trying to access files from Github via the SubscriptionDataSource. The idea behind this is to pipe some data like CSVs but that are hosted on Github rather than Dropbox. Because this specific data is private, I cannot make the repository that contains them public. For testing purposes I've created a dummy repo with dummy data.
I've attached a backtest and noticed a few things when attempting to pull the files from github.
- Including (uncommenting) the "headers" in the SubscriptionDataSource will make the code fail, on both public and private repositories.
- Fetching a private repository via the traditional Github API headers fails in SubscriptionDataSource, but succeeds when done via the self.Download() method with headers, for both public and private repositories. This leads me to believe that the SubscriptionDataSource headers are buggy.
I've put the file public at
https://github.com/marcdemers/test_fetchso that you could test it out too with your own Github token.
Of course the github token in the backtest is disabled.
Is this intended behavior? If so, what would be other possibilities to pipe data into the OnData method for thousands of tickers daily?
Tomas Rampas
Hi Marc,
I assume feeding data this way would slow down your backtest significantly.Â
For such scenario the best option is to have LEAN running directly on your machine or VM and having your custom data stored locally in the subfolder structure of the LEAN itself. But unfortunatly it needs some devops expertise...
Marc Demers
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!