Lean-cli local development, what data to subscribe?

Here's a guide to address Wang's concerns and questions:

1. Cloud IDE Autocomplete for Python: To enable autocomplete in the cloud IDE, ensure you include the necessary imports in your Python files. Use from AlgorithmImports import * at the top of your files to ensure all necessary modules are available for autocomplete.

2. Local Development without Data Subscription: While a data subscription is required for full access to historical data, you can still run backtesting scripts using sample data or custom data sources. Consider using publicly available datasets or creating custom data types for testing. You can refer to examples like the CustomDataRegressionAlgorithm for guidance on implementing custom data types.

3. Running Backtesting Scripts with Sample Data: Implement custom data types in your algorithm to simulate data feeds, allowing you to test strategies without a data subscription. This approach can help you develop and test your algorithms locally without incurring additional costs.

4. SPY and QQQ Historical Option Data: For SPY and QQQ historical option data, consider purchasing datasets like the US Index Options dataset by AlgoSeek, which covers European Option contracts for US indices. This dataset provides comprehensive historical data starting from January 2012.

5. Setting Up Local Autocomplete: If using local development environments like VS Code or PyCharm, ensure the correct extensions and configurations are set up. For VS Code, install the Python and Pylance extensions. For PyCharm, open the project directory and wait for indexing to complete. Ensure your project includes the necessary imports for autocomplete to function correctly.

6. Understanding Data Requirements: Review the QuantConnect documentation to understand the data requirements and limitations for local development. This will help you be aware of any necessary subscriptions or alternative data sources.

7. Explore Dataset Listings: Visit the QuantConnect datasets page to explore available datasets and their pricing, ensuring you select the appropriate data for your trading strategies. The US Index Options dataset by AlgoSeek is a good option for SPY and QQQ historical option data.

Sources: Costs - QuantConnect.com, Autocomplete - QuantConnect.com, US Index Options - QuantConnect.com, Crypto - QuantConnect.com, Documentation Home - QuantConnect.com

For type hint support with QuantConnect algorithm programming in Python, I wonder does the quantconnect-stubs package help? (It was mentioned in a readme under the QuantConnect lean sources at GithHub)

If one could try to put together an ad hoc walkthrough for local backtests with lean-cli, though tbh I'm new to this platform myself:

1) Create a cloud_root directory (directory name may vary, of course) and set up a python virtual environment there. One could use virtualenv though preferences may vary here. (A package management tool such as pdm or poetry could be used to create a python project in this dir, though the dir will contain other python projects shortly.)

2) Ensure that the docker service is installed and working. There's a small hello-world image available for purpose of test

3) Install the lean CLI tools in the python virtual environment: pip install lean

4) under the cloud_root directory, run ‘lean init’. This will create a ‘lean.json’ file and download some sample data for backtests under the local object store root dir, named ‘data’ (This data dir will be passed to Docker, at some point)

5) To create a Python project named ‘quantexample’ for backtesting, run ‘lean project-create -l python quantexample’ from within the cloud_root dir

6) After editing the example to suit, then to backtest locally under docker, run `lean backtest quantexample' from within the cloud_root dir. (Initially, this will download something more than 9 GB of docker image data. Reportedly, containerd-snapshotter can be enabled in docker, to allow for resumble downloads during docker pull and other features. Of course, docker will need to be restarted after this feature is enabled.) The backest cmd supports a number of additional args, available with `lean backtest --help'

Towards coding the trading algorithm, the LEAN sources contain a number of examples, illustrating some API features. Personally, I'd started with Greg Bland's QuantConnect 101. His example features a trading strategy based on a Bollinger Bands indicator available directly in QuantConnect. 

It seems the QuantConnect API has been updated in some ways since the article was published, e.g ‘self.set_warm_up(…)’ instead of ‘self.SetWarmUp(…)’

For local backtests with the LEAN docker image, the backtesting data provided from quant seems to be limited to some very old data (at least in  FOREX symbols) as old as 2014. At least for FOREX data, these are provided as zip files containing csv files. It may be possible to guess out the CSV format and write newer CSV files for backtesting with newer data, e.g via yfinance or directly via a broker's web API. For the data as provided, perhaps it may be enough for a simple example tho?

The ‘lean cloud’ commands would provide support for synching with the Lean cloud system, for backtest or live trades

HtH!