Local lean backtest/report produce incorrect results

Hi Alpha23

It could be owing to 2 reasons:

1. You are using a different version of LEAN.
2. You haven't had the data ready for local backtesting.

You might want to run the following commands to avoid these issues:

lean backtest "My Project" --download-data --update

These instruct the cli to pull the latest LEAN image for backtesting, and download necessary data during backtest. Alternatively, use the CLI downloader to prepare the data in advance. Make sure you have enough QCC before downloading data.

Best
Louis

I was able to run the this backtest locally on Dec 30.  I'm now receiving the following errors in the output log.  Why are there now 'Configuration key not found' issues and how to fix?


2023-01-09T20:29:42.9052562Z TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/
2023-01-09T20:29:42.9264851Z TRACE:: Config.Get(): Configuration key not found. Key: version-id - Using default value: 
2023-01-09T20:29:42.9265735Z TRACE:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: /Lean/Data
2023-01-09T20:29:42.9267168Z TRACE:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.5.0.0 Mode: DEBUG (64bit) Host: desktop
2023-01-09T20:29:42.9309605Z TRACE:: Engine.Main(): Started 8:29 PM
2023-01-09T20:29:42.9351287Z TRACE:: Config.Get(): Configuration key not found. Key: lean-manager-type - Using default value: LocalLeanManager
2023-01-09T20:29:42.9514822Z TRACE:: JobQueue.NextJob(): Selected /LeanCLI/main.py
2023-01-09T20:29:43.0254479Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-capacity - Using default value: 120
2023-01-09T20:29:43.0255136Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-time-interval-minutes - Using default value: 1440
2023-01-09T20:29:43.0255509Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-refill-amount - Using default value: 18
2023-01-09T20:29:43.0261653Z TRACE:: Config.GetValue(): storage-limit - Using default value: 10737418240
2023-01-09T20:29:43.0262100Z TRACE:: Config.GetValue(): storage-permissions - Using default value: 3
2023-01-09T20:29:43.0262447Z TRACE:: Config.Get(): Configuration key not found. Key: backtest-name - Using default value: local
2023-01-09T20:29:43.0299755Z TRACE:: Config.Get(): Configuration key not found. Key: job-organization-id - Using default value: 
2023-01-09T20:29:43.0300400Z TRACE:: Config.Get(): Configuration key not found. Key: python-venv - Using default value: 
2023-01-09T20:29:43.0305673Z TRACE:: Config.Get(): Configuration key not found. Key: data-permission-manager - Using default value: DataPermissionManager
2023-01-09T20:29:43.0401819Z TRACE:: Config.Get(): Configuration key not found. Key: job-organization-id - Using default value: 
2023-01-09T20:29:43.0409141Z TRACE:: Config.GetValue(): data-purchase-limit - Using default value: 79228162514264337593543950335
2023-01-09T20:29:43.6272200Z TRACE:: ApiDataProvider(): Will use organization Id 'd006140812df925ef58c19585950ab7e'.
 

Hi Alpha23

LEAN will read data directory in the config.json setting from its launcher. The line of log only tells you that if you didn’t specify one, it would use the default (“../../../data”) automatically. So, you were not receiving any error message from there. You don’t need to worry about it since docker will mount your data folder in the cli directory into your docker container.

Please check my previous comment for your issue,

Best
Louis

The CLI shows similar information as the log, both below, and results in an Unhandled Exception in the CLI (see below).  How do I fix?  

CLI:

lean backtest project2 --download-data --update
Configure credentials for QuantConnect
Pulling quantconnect/lean:latest...
latest: Pulling from quantconnect/lean
Digest: sha256:636cc90a61f06fa7fa22f0093d08f6f28d83055250ebd0e3c2afb40bd7448b01
Status: Image is up to date for quantconnect/lean:latest
docker.io/quantconnect/lean:latest
20230112 00:31:10.147 TRACE:: Config.Get(): Configuration key not found. Key: plugin-directory - Using default value:
20230112 00:31:10.170 TRACE:: Composer(): Loading Assemblies from /Lean/Launcher/bin/Debug
20230112 00:31:10.200 TRACE:: Python for .NET Assembly: Python.Runtime, Version=2.0.17.0, Culture=neutral, PublicKeyToken=5000fea6cba702dd
20230112 00:31:10.243 TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/
20230112 00:31:10.258 TRACE:: Config.Get(): Configuration key not found. Key: version-id - Using default value:
20230112 00:31:10.258 TRACE:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: /Lean/Data
20230112 00:31:10.258 TRACE:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.5.0.0 Mode: DEBUG (64bit) Host: desktop2
20230112 00:31:10.297 TRACE:: Engine.Main(): Started 12:31 AM
20230112 00:31:10.325 TRACE:: Config.Get(): Configuration key not found. Key: lean-manager-type - Using default value: LocalLeanManager
20230112 00:31:10.356 TRACE:: JobQueue.NextJob(): Selected /LeanCLI/main.py
20230112 00:31:10.440 TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-capacity - Using default value: 120
20230112 00:31:10.440 TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-time-interval-minutes - Using default value: 1440
20230112 00:31:10.440 TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-refill-amount - Using default value: 18
20230112 00:31:10.441 TRACE:: Config.GetValue(): storage-limit - Using default value: 10737418240
20230112 00:31:10.441 TRACE:: Config.GetValue(): storage-permissions - Using default value: 3
20230112 00:31:10.441 TRACE:: Config.Get(): Configuration key not found. Key: backtest-name - Using default value: local
20230112 00:31:10.446 TRACE:: Config.Get(): Configuration key not found. Key: data-channel - Using default value:
20230112 00:31:10.446 TRACE:: Config.Get(): Configuration key not found. Key: job-organization-id - Using default value:
20230112 00:31:10.446 TRACE:: Config.Get(): Configuration key not found. Key: python-venv - Using default value:
20230112 00:31:10.447 TRACE:: Config.Get(): Configuration key not found. Key: data-permission-manager - Using default value: DataPermissionManager
20230112 00:31:10.460 TRACE:: Config.Get(): Configuration key not found. Key: job-organization-id - Using default value:
20230112 00:31:10.461 TRACE:: Config.GetValue(): data-purchase-limit - Using default value: 79228162514264337593543950335
20230112 00:31:11.400 TRACE:: ApiDataProvider(): Will use organization Id 'd006140812df925ef58c19585950ab7e'.
Unhandled exception. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
---> System.InvalidOperationException: ApiDataProvider(): Must agree to terms at 
https://www.quantconnect.com/terms/data/?organization=d006140812df925ef58c19585950ab7e, before using the ApiDataProvider
  at QuantConnect.Lean.Engine.DataFeeds.ApiDataProvider..ctor() in 
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Engine/DataFeeds/ApiDataProvider.cs:line 107
  at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
  --- End of inner exception stack trace ---
  at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
  at System.Activator.CreateInstance(Type type, Boolean nonPublic, Boolean wrapExceptions)
  at QuantConnect.Util.Composer.GetExportedValueByTypeName[T](String typeName, Boolean forceTypeNameOnExisting) in 
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Common/Util/Composer.cs:line 281
  at QuantConnect.Lean.Engine.LeanEngineAlgorithmHandlers.FromConfiguration(Composer composer, Boolean researchMode) in 
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Engine/LeanEngineAlgorithmHandlers.cs:line 221
  at QuantConnect.Lean.Engine.Initializer.GetAlgorithmHandlers(Boolean researchMode) in 
/LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Engine/Initializer.cs:line 78
  at QuantConnect.Lean.Launcher.Program.Main(String[] args) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Launcher/Program.cs:line 71
Error: Something went wrong while running 'project2' in the 'backtesting' environment, the output is stored in 
'project2/backtests/2023-01-11_17-31-07'


LOG:

2023-01-12T00:31:10.2437836Z TRACE:: Config.Get(): Configuration key not found. Key: data-directory - Using default value: ../../../Data/
2023-01-12T00:31:10.2582528Z TRACE:: Config.Get(): Configuration key not found. Key: version-id - Using default value: 
2023-01-12T00:31:10.2583314Z TRACE:: Config.Get(): Configuration key not found. Key: cache-location - Using default value: /Lean/Data
2023-01-12T00:31:10.2585165Z TRACE:: Engine.Main(): LEAN ALGORITHMIC TRADING ENGINE v2.5.0.0 Mode: DEBUG (64bit) Host: desktop2
2023-01-12T00:31:10.2977980Z TRACE:: Engine.Main(): Started 12:31 AM
2023-01-12T00:31:10.3258495Z TRACE:: Config.Get(): Configuration key not found. Key: lean-manager-type - Using default value: LocalLeanManager
2023-01-12T00:31:10.3566996Z TRACE:: JobQueue.NextJob(): Selected /LeanCLI/main.py
2023-01-12T00:31:10.4404633Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-capacity - Using default value: 120
2023-01-12T00:31:10.4405269Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-time-interval-minutes - Using default value: 1440
2023-01-12T00:31:10.4405556Z TRACE:: Config.GetValue(): scheduled-event-leaky-bucket-refill-amount - Using default value: 18
2023-01-12T00:31:10.4413735Z TRACE:: Config.GetValue(): storage-limit - Using default value: 10737418240
2023-01-12T00:31:10.4414199Z TRACE:: Config.GetValue(): storage-permissions - Using default value: 3
2023-01-12T00:31:10.4414629Z TRACE:: Config.Get(): Configuration key not found. Key: backtest-name - Using default value: local
2023-01-12T00:31:10.4463374Z TRACE:: Config.Get(): Configuration key not found. Key: data-channel - Using default value: 
2023-01-12T00:31:10.4463967Z TRACE:: Config.Get(): Configuration key not found. Key: job-organization-id - Using default value: 
2023-01-12T00:31:10.4464403Z TRACE:: Config.Get(): Configuration key not found. Key: python-venv - Using default value: 
2023-01-12T00:31:10.4471546Z TRACE:: Config.Get(): Configuration key not found. Key: data-permission-manager - Using default value: DataPermissionManager
2023-01-12T00:31:10.4604308Z TRACE:: Config.Get(): Configuration key not found. Key: job-organization-id - Using default value: 
2023-01-12T00:31:10.4614498Z TRACE:: Config.GetValue(): data-purchase-limit - Using default value: 79228162514264337593543950335
2023-01-12T00:31:11.4006720Z TRACE:: ApiDataProvider(): Will use organization Id 'd006140812df925ef58c19585950ab7e'.
 

Hi Alpha23

The problem comes from here:

---> System.InvalidOperationException: ApiDataProvider(): Must agree to terms at 
https://www.quantconnect.com/terms/data/?organization=d006140812df925ef58c19585950ab7e, before using the ApiDataProvider

You need to agree the terms of use of downloading local data through cli in prior. Simply login your account in QC cloud and go to the link in the error message. After agreeing the terms, the job shall be carried out normally if you have enough QCC.

Best
Louis

Thank you for the clarification.  When I initially ran ‘lean init', sample data was populated.  For testing purposes, can the sample data - which included the entire testing period in my backtest above for the symbol being tested - be used and if yes, how do I get around errors and use the sample data?

Hi Alpha23

Sample data was not gauranteed with accuracy. You may use it to test the functionality of your algorithms, but not recommended on trading purpose. Please download the needed data via the cli downloader, or prepare your own custom data.

Best
Louis

I'm using 1 symbol, AAPL, to test my algorithm locally.  I've manually checked that there is data for AAPL over the backtest period.  I've also tested with SPY with the same algorithm with the same results.  How can I test the functionality of my algorithms locally if the sample data is completely inaccurate? The test results for the algorithm executed locally is not just incorrect by a few trades?  The results are completely wrong wherein there are no trades are executed after 2014.  I suspect that the issue does not have to do with the data but rather the lean docker container.

My concern is that the local lean docker container does not work correctly and produces completely different results than the on-line version.  I would like to verify/test my algorithm locally before spending money to download data locally only to have the same erroneous results.  

Hi Alpha23

We have tested the consistency of cloud and cli environment to ensure the functionality extends. So if your codes are the same while using the updatest version of the cli and docker image, the results should match. We are not able to provide sample real local data due to the agreement with our data providers. Hope this can ease your concern.

Best
Louis

I'm confused as to why it's called sample data, labeled as a specific ticker which includes data over a date range, and then stated that this “sample data" is for testing purposes.  A consequence of your statements above is that there is no sample data, despite it being called this, because any results are grossly erroneous which means that nothing is actually being tested.  In order to verify that your engine works locally, customers will need to purchase data.

I attempted to download the one ticker, AAPL, to test and received the following error:

lean data download --dataset "US Equities" --organization "<ORGANIZATION_ID>" --data-type "Trade" --ticker "AAPL" --resolution "Daily" --end "20230117" 
Error: Your organization needs to have an active Security Master subscription to download data from the 'US Equities' dataset
You can add the subscription at https://www.quantconnect.com/datasets/quantconnect-security-master/pricing


The pricing suggests $420 for a subscription.  Is it true that I will need to pay $420, plus $1 for each ticker (I actually only need to test on 1 so the per ticker is not an issue), to locally test just to verify that quantconnect is a platform that I should invest in (given that I will only be using it locally)?

Hi alpha23 yes the pricing can get tricky for local data. I think the sample AAPL data should be ok but you might want to check a few random days data to confirm. Here's a thread  on using local data/ pricing:

https://www.quantconnect.com/forum/discussion/14609/my-notes-on-payment-for-local-historical-data-using-lean-cli/p1

Non Compete Your link was very helpful; the data pricing is unclear for local data.  The source of my problem ended up being that the local data was only complete with daily data and the algorithm was using minute data.  Setting the equity resolution to daily produced expected results.  The sample data does appear to be slightly scrambled but is good enough for testing.