Hello, 

Here are my notes on using local data. I found the pricing structure and process confusing and spent money unnecessarily so hopefully this will be helpful to others.

1 - When you purchase products or data you will be asked to re-purchase your current Seat (Reasearcher for example) and any other required subscriptions again for the time period you are selecting, with any previous purchases that overlap that time period prorated.  For example if you go to purchase the AlgoSeek Bulk ( https://www.quantconnect.com/datasets/algoseek-us-equities/pricing ) history you will be asked to purchase US Equity Securities Master ( https://www.quantconnect.com/datasets/quantconnect-us-equity-security-master ) and your current seat type for the same time period. I assume this is because your seat and US Equity Securities Master are required go the Bulk AlgoSeek purchase to be functional.

 

2 - In order to get any Quantconnect data locally you will need to purchase US Equity Securities Master  ( https://www.quantconnect.com/datasets/quantconnect-us-equity-security-master ). You should see an error message about it missing if you try “lean backtest” or “lean research” without it. After purchasing US Equity Securities Master you can use QCC to purchase individual stock data. 

 

3 - You can purchase QCC on your organization dashboard ( https://www.quantconnect.com/organization ) from the “Purchase Credit” link. QCC charges show under “Manage billing” link on the same page .

 

3 - “lean research” CLI command will charge you QCC for each run, for each equity. For example if you use 2 stocks' daily data, you will be charged 200 QCC each time you run “lean research” . This can be controlled using “-data-purchase-limit 0 --data-provider Local”. 

 

4 - “lean backtest” CLI command will charge you any time there is new data for your equities. For example if your backtest uses 2 stocks' daily data and you downloaded the data yesterday at a cost of 200 QCC, when you run a backtest again today it will download today's data (1 extra line per stock data file) and charge you 200 QCC again. If you run “lean backtest” again, you wont be charged again until tomorrow.  This can be controlled using “--data-purchase-limit 0 ” so you can prevent the download of data if you like.  If you would like to download all the necessary data for your backtest without concerns to cost, you can use the --download-data option  ( https://github.com/QuantConnect/lean-cli  ).

 

5 - In order to download data new equities, you can use the “lean data download” command. I currently don't see a way to only download new equities and not be charged for yesterday's again if you figure out which equities you need within your backtest. You need to download the data outside of the backtest and run the backtest using “--data-purchase-limit 0 ” to prevent yourself from being re-charged for stocks you already have a previous day's data for, if you don't need the more recent data.

6 - You might need to use “--overwrite” with “lean data download” if you already have existing local data, but the data doesn't cover the days you are looking for. In that case it will look like the history is blank in your backtest.

 

7 - I haven't purchased it yet to confirm , but in order to avoid using QCC for data, you can purchase this AlgoSeek Bulk History for daily/hourly data as seen here:

https://www.quantconnect.com/datasets/algoseek-us-equities/pricing 

According to support:

The bulk data is a one-time purchase, but you will have to "re-purchase" it again in one year to unlock the updates since the updates subscription expires in one year.

The bulk data should give you updated daily/hourly data without needing to spend any QCC on downloading data.