Running a local research environment?

Ok, so I made a little bit of progress with this. I installed Jupyter and pythonnet locally, then used the QuantBook template from the online environment as inspiration. I can load the .net libraries, but when I try to create a new instance of QuantBook, I get the following error:

QuantBook.Main(): System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Data\market-hours\market-hours-database.json'.

It would appear that some data files are necessary for this to run. Where can I find them?

%matplotlib inline
import sys
sys.path.append(r"E:\programmieren\LEAN\Lean\Jupyter\bin\Debug")

from clr import AddReference
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Jupyter")
AddReference("QuantConnect.Indicators")
from System import *
from QuantConnect import *
from QuantConnect.Data.Market import TradeBar, QuoteBar
from QuantConnect.Jupyter import *
from QuantConnect.Indicators import *
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from QuantConnect.Data.Fundamental import IncomeStatement

# Create an instance
qb = QuantBook()

direct link to the file

Ok, thanks, so it's in the repository. I actually made some more progress by pointing the config.json file in the QuantConnect.Jupyter project to the correct location like this:

I also modified the notebook to change the working directory to the location where the Jupyter project is builiding

Now I can at least use the classes. Unfortunately, when trying to get data, I only get an empty dataframe

Can this be hooked up to the QC data through the API somehow or is this as far as I'll get? I tried to supply the same handlers as in the backtest configuration, but I couldn't get it to do anything. I'll see if I can set up a local project to debug it from C# instead of using the notebook.

By the way: how did you manage to get pythonnet to resolve the Resolution? function arguments correctly? I was only able to do it with an explicit cast.

As it turns out, I misunderstood how the backtesting works. It's loading data from the file system, so you actually need to download this from somewhere. I set up a debug project so I could get command line output from the QuantBook object and it complained about some missing files in the data folder, which, apart from some example data, is empty.

So for a local research environment, you'll need to get the data from somewhere else. Maybe someone from QC would be nice enough to provide assistance with how to get the file format right as the data seems to be stored as csvs in zip files.

As a proof of concept, I proceeded to download some forex data from FXCM via the QC data page. I can load it from my test program like this

Which prints out some data. Unfortunately, trying to replicate it in the notebook, I get the following error:

trying to call this:

Playing around a little with error messages (I haven't been able to find out how to debug when triggering code from the notebook, attaching to python.exe didn't work) I found that the error apparently happens in the function 

more specifically on the last line in the return statement in the ToArray function.

Since my original goal of messing with options is probably not gonna happen in the local environment for me as I don't have enough cash to burn for an adequate data provider, I will probably leave it at that, but I'd still be interested in getting at least the proof-of-concept to work just for kicks.

Stefan Reutter"how did you manage to get pythonnet to resolve the Resolution?"
You need to install QuantConnect's version of pythonnet.

Alexandre Catarino, thank you. I see. Do you expect that this would also fix the conversion error? I'll give it a whirl when I have some spare time later this week.

Hi Stefan Reutter, thank you for putting this up! I'm getting started with QC and cloned the Basic Template to run locally. I have a working python distribution with jupyter notebook installed. But I have never used pythonnet to import dll files into python.

After installing pythonnet and running the first cell, I got this error.

By the way I found this python package quantconnect that said it was developed by QC python team. Can any body tell me what it is for and how to use it?

Thanks

So I was too eager to setup a local research environment and missed some details in the comments #8773 and #8774 above. In which, Stefan already pointed out that this is not an efficient way because the engine loads data from file system and one needs to manually download data for backtesting locally.

I was able to import the engine by following comment #8773 anyway. The exact steps are:

1. Changing plugin path in config.json, 

2. Rebuilding the QuantConnect.Jupyter project and 

3. Inserting the path into jupyter notebook using

.

Hi!

I followed the document and setup everything. library on each python side or c# side loaded. Works well.

But I got an error when I try to create a QuantBook instance.

Any Ideas? Is there any addtional parameters I need to give at config.json? 

I am using 

Windows 10

python 3.6

Thanks!

Is anyone currently running the research environment locally and solved their issues above?

It would be an useful option for model training as training on substantial data sets obviously takes a lot of time.

@Arima this means it could not locate the LEAN binaries. You just need to reference LEAN / put the binaries into your Jupyter folder where it can be found.

Hi Jared Broad are you able to elaborate on this? I am getting a similar error, but I don't exactly know what I need to do to fix this / where do I reference the binaries 

Hi,

If we want to run jupyter in the directory with the binaries (e.g. Lean/Laucher/bin/Debug), we just need to uncomment  the "composer-dll-directory" key of the config file that has "." (it means, the current directory) as default. 
In my PC, I got:

Note that my data-folder is pointing to a relative path in this case.

Alternatively, we can run jupyter in any other directory of our computer with the following settings:

in this case, we also need to add the location of the binaries to the path in the first lines of the notebook script:

For more information, please check out the readme in QuantConnect/Lean repo.

This is the place to look now i think: