Share New Research 
Hi folks. I'm having the hardest time figuring out how to configure the data generation directory. I'm running:
lean data generate --start=20200601 --symbol-count=1 --security-type=Equity
I'm running this from my project diretory. The directory I lean pulled all my projects into for local development. It has a ./data sub-directory. I'd like the generated data to go into that directory, so I can do some local development / debugging / testing.
I keep seeing this error when running the above command:
20210412 21:35:07.695 TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/
Where does the "data-directory" key live? I tried to put it in my lean.json, which also has "data-folder" key, but that didn't work. I tried to put it in one of my project's config.json, that didn't work either. Is there some master config.json that I'm missing? Thanks!
Jasper van Merle
Hi Michael,
The "Configuration key not found. Key: data-directory" message is not an error, you don't need to worry about that. If the lean data generate command fails with a "Something went wrong while running the random data generator, see the logs above for more information" message, there is most likely a line somewhere near the bottom of the logs saying what went wrong. If you post the full logs I can help you find it.
The data directory that is used is configured in the data-folder property in the lean.json file, which must be a path relative to the lean.json file. By default it is set to "data", which points to the ./data directory that was generated when you ran lean init. If you set it to something else, make sure you set it to a path to a valid LEAN data directory (i.e. pointing it to an empty directory will not work).
Michael P
Thanks for the help Jasper! I re-ran and let it complete, checked the data directory and it all looks good. The generated files are in there. Nice!
Michael P
Hi Jasper,
Follow-up. I tried running a local backtest, but it appears it is not finding the data files. It's using the path I mentioned above. I think something is misconfigured somewhere, but haven't been able to figure it out. Here's the log and thanks!
2021-04-13T00:44:30.5548150Z TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/ 2021-04-13T00:44:30.5657440Z TRACE:: Config.Get(): Configuration key not found. Key: version-id - Using default value: 2021-04-13T00:44:30.5672120Z TRACE:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: /Lean/Data 2021-04-13T00:44:30.5686350Z TRACE:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.5.0.0 Mode: DEBUG (64bit) 2021-04-13T00:44:30.5869170Z TRACE:: Engine.Main(): Started 12:44 AM 2021-04-13T00:44:30.5929840Z TRACE:: Config.GetValue(): job-project-id - Using default value: 0 2021-04-13T00:44:30.6872640Z TRACE:: Config.GetValue(): regression-update-statistics - Using default value: False 2021-04-13T00:44:30.6963140Z TRACE:: Config.Get(): Configuration key not found. Key: lean-manager-type - Using default value: LocalLeanManager 2021-04-13T00:44:30.9282730Z TRACE:: JobQueue.NextJob(): Selected /LeanCLI/main.py 2021-04-13T00:44:31.1745960Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-capacity - Using default value: 120 2021-04-13T00:44:31.1757530Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-time-interval-minutes - Using default value: 1440 2021-04-13T00:44:31.1779150Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-refill-amount - Using default value: 18 2021-04-13T00:44:31.1965640Z TRACE:: Config.Get(): Configuration key not found. Key: algorithm-id - Using default value: main 2021-04-13T00:44:31.2347490Z TRACE:: Config.Get(): Configuration key not found. Key: data-permission-manager - Using default value: DataPermissionManager 2021-04-13T00:44:31.3319820Z TRACE:: Config.GetValue(): api-data-update-period - Using default value: 5 2021-04-13T00:44:31.3457010Z TRACE:: Config.Get(): Configuration key not found. Key: object-store-root - Using default value: ./storage 2021-04-13T00:44:31.3698920Z TRACE:: AlgorithmManager.CreateTokenBucket(): Initializing LeakyBucket: Capacity: 120 RefillAmount: 18 TimeInterval: 1440 2021-04-13T00:44:31.3917340Z TRACE:: Config.GetValue(): algorithm-manager-time-loop-maximum - Using default value: 20 2021-04-13T00:44:31.4121670Z TRACE:: TextSubscriptionDataSourceReader.SetCacheSize(): Setting cache size to 71582788 items 2021-04-13T00:44:32.4048110Z TRACE:: Config.GetValue(): algorithm-creation-timeout - Using default value: 90 2021-04-13T00:44:32.4187910Z TRACE:: DebuggerHelper.Initialize(): initializing python... 2021-04-13T00:44:32.4202710Z TRACE:: PythonInitializer.Initialize(): start... 2021-04-13T00:44:34.2682500Z TRACE:: PythonInitializer.Initialize(): ended 2021-04-13T00:44:34.2693580Z TRACE:: DebuggerHelper.Initialize(): python initialization done 2021-04-13T00:44:34.2702610Z TRACE:: DebuggerHelper.Initialize(): starting... 2021-04-13T00:44:34.2711470Z TRACE:: DebuggerHelper.Initialize(): waiting for PTVSD debugger to attach at localhost:5678... 2021-04-13T00:44:48.8415750Z TRACE:: DebuggerHelper.Initialize(): started 2021-04-13T00:44:48.8599090Z TRACE:: AlgorithmPythonWrapper(): Python version 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:25:33) [GCC 7.3.0]: Importing python module main 2021-04-13T00:44:52.3539680Z TRACE:: AlgorithmPythonWrapper(): main successfully imported. 2021-04-13T00:44:52.3723230Z TRACE:: AlgorithmPythonWrapper(): Creating IAlgorithm instance. 2021-04-13T00:44:53.2246970Z TRACE:: LocalObjectStore.Initialize(): Storage Root: /Lean/Launcher/bin/Debug/storage/QCAlgorithm 2021-04-13T00:44:53.2819330Z TRACE:: Config.Get(): Configuration key not found. Key: security-data-feeds - Using default value: 2021-04-13T00:44:53.6247580Z TRACE:: BaseSetupHandler.SetupCurrencyConversions(): Symbol Quantity Conversion = Value in USD USD: $ 10000.00 @ 1.00 = $10000.0 ------------------------------------------------- CashBook Total Value: $10000.0 2021-04-13T00:44:53.6392610Z TRACE:: BacktestingResultHandler(): Sample Period Set: 69.49 2021-04-13T00:44:53.6455980Z TRACE:: Config.GetValue(): forward-console-messages - Using default value: True 2021-04-13T00:44:53.6526950Z TRACE:: JOB HANDLERS: 2021-04-13T00:44:53.6535770Z TRACE:: DataFeed: QuantConnect.Lean.Engine.DataFeeds.FileSystemDataFeed 2021-04-13T00:44:53.6547060Z TRACE:: Setup: QuantConnect.Lean.Engine.Setup.ConsoleSetupHandler 2021-04-13T00:44:53.6554460Z TRACE:: RealTime: QuantConnect.Lean.Engine.RealTime.BacktestingRealTimeHandler 2021-04-13T00:44:53.6563600Z TRACE:: Results: QuantConnect.Lean.Engine.Results.BacktestingResultHandler 2021-04-13T00:44:53.6572220Z TRACE:: Transactions: QuantConnect.Lean.Engine.TransactionHandlers.BacktestingTransactionHandler 2021-04-13T00:44:53.6589470Z TRACE:: Alpha: QuantConnect.Lean.Engine.Alphas.DefaultAlphaHandler 2021-04-13T00:44:53.6597210Z TRACE:: ObjectStore: QuantConnect.Lean.Engine.Storage.LocalObjectStore 2021-04-13T00:44:53.6605970Z TRACE:: History Provider: QuantConnect.Lean.Engine.HistoricalData.SubscriptionDataReaderHistoryProvider 2021-04-13T00:44:53.7387180Z TRACE:: AlgorithmManager.Run(): Begin DataStream - Start: 10/1/2020 12:00:00 AM Stop: 4/12/2021 12:44:52 AM 2021-04-13T00:44:53.7393230Z TRACE:: Debug: Launching analysis for main with LEAN Engine v2.5.0.0 2021-04-13T00:44:53.9307160Z TRACE:: Config.GetValue(): data-feed-workers-count - Using default value: 4 2021-04-13T00:44:53.9325830Z TRACE:: Config.GetValue(): data-feed-max-work-weight - Using default value: 400 2021-04-13T00:44:53.9337100Z TRACE:: Config.Get(): Configuration key not found. Key: data-feed-queue-type - Using default value: QuantConnect.Lean.Engine.DataFeeds.WorkScheduling.WorkQueue, QuantConnect.Lean.Engine 2021-04-13T00:44:53.9351640Z TRACE:: WeightedWorkScheduler(): will use 4 workers and MaxWorkWeight is 400. Queue type: WorkQueue 2021-04-13T00:44:54.0222670Z TRACE:: Config.GetValue(): show-missing-data-logs - Using default value: False 2021-04-13T00:44:54.0490480Z TRACE:: ApiDataProvider.Fetch(): Attempting to get data from QuantConnect.com's data library for symbol(EON), resolution(Hour) and date(1/1/0001). 2021-04-13T00:44:55.4730140Z ERROR:: ApiDataProvider.Fetch(): Unable to remotely retrieve data for path /Lean/Data/equity/usa/hour/eon.zip. Please make sure you have the necessary data in your online QuantConnect data library. 2021-04-13T00:44:55.5217440Z TRACE:: ApiDataProvider.Fetch(): Attempting to get data from QuantConnect.com's data library for symbol(SPY), resolution(Hour) and date(1/1/0001). 2021-04-13T00:44:55.5251840Z TRACE:: UniverseSelection.AddPendingInternalDataFeeds(): Adding internal benchmark data feed SPY,SPY,Hour,TradeBar,Trade,Adjusted,Internal 2021-04-13T00:44:55.6652660Z ERROR:: ApiDataProvider.Fetch(): Unable to remotely retrieve data for path /Lean/Data/equity/usa/hour/spy.zip. Please make sure you have the necessary data in your online QuantConnect data library. 2021-04-13T00:44:55.8282950Z TRACE:: ApiDataProvider.Fetch(): Attempting to get data from QuantConnect.com's data library for symbol(EON), resolution(Hour) and date(1/1/0001). 2021-04-13T00:44:55.9671910Z ERROR:: ApiDataProvider.Fetch(): Unable to remotely retrieve data for path /Lean/Data/equity/usa/hour/eon.zip. Please make sure you have the necessary data in your online QuantConnect data library. 2021-04-13T00:44:56.0355190Z TRACE:: Synchronizer.GetEnumerator(): Exited thread. 2021-04-13T00:44:56.0394500Z TRACE:: AlgorithmManager.Run(): Firing On End Of Algorithm... 2021-04-13T00:44:56.0512280Z TRACE:: Engine.Run(): Exiting Algorithm Manager 2021-04-13T00:44:56.0545460Z TRACE:: Debug: HELLO WORLD! 2021-04-13T00:44:56.0578170Z TRACE:: Debug: Algorithm Id:(main) completed in 2.39 seconds at 0k data points per second. Processing total of 1 data points. 2021-04-13T00:44:56.0645900Z TRACE:: FileSystemDataFeed.Exit(): Start. Setting cancellation token... 2021-04-13T00:44:56.0679720Z TRACE:: FileSystemDataFeed.Exit(): Exit Finished. 2021-04-13T00:44:56.0704910Z TRACE:: DefaultAlphaHandler.Exit(): Exiting... 2021-04-13T00:44:56.0903510Z TRACE:: DefaultAlphaHandler.Exit(): Ended 2021-04-13T00:44:56.0931140Z TRACE:: BacktestingResultHandler.Exit(): starting... 2021-04-13T00:44:56.0951050Z TRACE:: BacktestingResultHandler.Exit(): Saving logs... 2021-04-13T00:44:56.1126530Z TRACE:: Debug: Your log was successfully created and can be retrieved from: /Results/main-log.txt 2021-04-13T00:44:56.1142480Z TRACE:: BacktestingResultHandler.Run(): Ending Thread... 2021-04-13T00:44:56.1171710Z TRACE:: StopSafely(): waiting for 'Result Thread' thread to stop... 2021-04-13T00:44:56.4487450Z TRACE:: STATISTICS:: Total Trades 0 STATISTICS:: Average Win 0% STATISTICS:: Average Loss 0% STATISTICS:: Compounding Annual Return 0% STATISTICS:: Drawdown 0% STATISTICS:: Expectancy 0 STATISTICS:: Net Profit 0% STATISTICS:: Sharpe Ratio 0 STATISTICS:: Probabilistic Sharpe Ratio 0% STATISTICS:: Loss Rate 0% STATISTICS:: Win Rate 0% STATISTICS:: Profit-Loss Ratio 0 STATISTICS:: Alpha 0 STATISTICS:: Beta 0 STATISTICS:: Annual Standard Deviation 0 STATISTICS:: Annual Variance 0 STATISTICS:: Information Ratio 0 STATISTICS:: Tracking Error 0 STATISTICS:: Treynor Ratio 0 STATISTICS:: Total Fees $0.00 STATISTICS:: Estimated Strategy Capacity $0 STATISTICS:: Fitness Score 0 STATISTICS:: Kelly Criterion Estimate 0 STATISTICS:: Kelly Criterion Probability Value 0 STATISTICS:: Sortino Ratio 0 STATISTICS:: Return Over Maximum Drawdown 0 STATISTICS:: Portfolio Turnover 0 STATISTICS:: Total Insights Generated 0 STATISTICS:: Total Insights Closed 0 STATISTICS:: Total Insights Analysis Completed 0 STATISTICS:: Long Insight Count 0 STATISTICS:: Short Insight Count 0 STATISTICS:: Long/Short Ratio 100% STATISTICS:: Estimated Monthly Alpha Value $0 STATISTICS:: Total Accumulated Estimated Alpha Value $0 STATISTICS:: Mean Population Estimated Insight Value $0 STATISTICS:: Mean Population Direction 0% STATISTICS:: Mean Population Magnitude 0% STATISTICS:: Rolling Averaged Population Direction 0% STATISTICS:: Rolling Averaged Population Magnitude 0% STATISTICS:: OrderListHash d41d8cd98f00b204e9800998ecf8427e 2021-04-13T00:44:56.4510080Z TRACE:: BacktestingResultHandler.SendAnalysisResult(): Processed final packet 2021-04-13T00:44:56.4577140Z TRACE:: Engine.Run(): Disconnecting from brokerage... 2021-04-13T00:44:56.4602540Z TRACE:: Engine.Run(): Disposing of setup handler... 2021-04-13T00:44:56.4621740Z TRACE:: Engine.Main(): Analysis Completed and Results Posted. 2021-04-13T00:44:56.4657160Z TRACE:: Engine.Main(): Packet removed from queue: main 2021-04-13T00:44:56.4687700Z TRACE:: LeanEngineSystemHandlers.Dispose(): start... 2021-04-13T00:44:56.4723410Z TRACE:: LeanEngineSystemHandlers.Dispose(): Disposed of system handlers. 2021-04-13T00:44:56.4742510Z TRACE:: LeanEngineAlgorithmHandlers.Dispose(): start... 2021-04-13T00:44:56.4785360Z TRACE:: LeanEngineAlgorithmHandlers.Dispose(): Disposed of algorithm handlers.Michael P
Maybe it's this:
"data-provider": "QuantConnect.Lean.Engine.DataFeeds.ApiDataProvider",
Think it should be:
"data-provider": "QuantConnect.Lean.Engine.DataFeeds.DefaultDataProvider",
Jasper van Merle
Hi Michael,
Yes, that should solve the issue. The ApiDataProvider only provides access to data in your QuantConnect Data Library, it does not fall back on local data in case the requested data cannot be found there.
Michael P
It worked! Now to figure out why VSCode debugger isn't hitting my breakpoints.
Michael P
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!