Avoiding Vendor Lock-In: Running LEAN On Your Server

Jared,

Like probably many other once-burned twice-shy Quantopian users, I would like to have a backup server running the open source version of QC "just in case". In all likelikhood I will rarely use it, since - as you correctly point out - it will be very laborious to keep it up to date with all the public data sources, etc.

I would describe myself at at level 4 on a scale of 10 in managing a linux server. What I don't have in expertise I make up for in sheer persistence and grit. I would like to host your Quantconnect Lean solution on Google Cloud Services, and have just signed up for an account. I was previously hosting  with a virtual server at ixwebhosting. But at $75 a month I realized it would be cheaper to go with Google, and probably more scalable. I quickly was able to set up a LAMP configuration and get the basics going on a Wordpress file, so I'm comfortable I'll be able to port my entire web hosting to Google. So now on to Quantconnect...

If my questions seem a bit ignorant, bear with me.  I'm sure among the Quantopian users there are a few thousand with similar concerns/ questions. I'm not familiar with Docker, and spent a good hour reading up on it and hoped to be able to  "launch" QC install with one command. This may be possible but I failed miserably. The best I achieved was to create a cluster with my project_id, create a directory to hold the QC files, and upload these to that directory using the command:

git clone https://github.com/QuantConnect/Lean

From there I played around a bit, but did not get far.

Does any documentation exist on setting up QC with AWS or Google Cloud? I searched through all the documentation on github and quantconnect but could find no other descriptionis of this process.

I would like to run QC using the python libraries.

Thanks

Hi Serge d'Adesky - the documentation for installing and running lean is ReadMe.md -- on the front page of the github repo.  

Get it while it's hot? :)

ReadME.md

Thanks Jared. I had already looked in there. So far, this is what I have found:

Google Cloud does not seem to support Mono.  At least in googling this I cannot find any suggestion that they do.  I am mentioning that in case others are thinking Google Cloud offers a reliable solution. 

I then went to my normal isp (ixwebhosting) where I have a virtual dedicated server running Centos 6.7.  There the instructions given on the Mono site for CentOS also fail. I think it has to do with the proxy settings in my /etc/yum.conf files,preventing EPEL from working properly. I am taking that up with hosting support.

Is it simpler in Amazon's AWS?

All cloud hosts support Mono, but you need to install it: http://www.mono-project.com/docs/getting-started/install

This is done for you in the docker file. I would recommend you learn how to use docker instead of installing each component piecemeal - 

Jared, 

Thanks for your response. Unfortunately it is not quite so simple with Docker if you are using CentOs 

At my centos provider : I was not able to load Nuget. At first I had problems installing Mono itself,, which does not work "out of the box" with the instructions provided on your link. But I did get a successful install after removing duplicates. .At least it gave me a "success" message with no warnings or fatal errors about missing dependencies. However, when I tried to test it on the hello.cs file, it bombed. 

root@cloud [~]# mcs hello.cs
hello.cs(7,9): error CS0103: The name `Debug' does not exist in the current context
Compilation failed: 1 error(s), 0 warnings
root@cloud [~]# cat hello.cs

public class HelloWorld
{
    static public void Main ()
    {
        Debug.WriteLine ("Hello Mono World");
    }
}
root@cloud [~]#
 

I spent a lot of time learnings Docker . But once I did I understood that it would install a file repository that was guaranteed to have all necessary file dependencies, and automate the process of installing these: However, when I look at the Docker code syntax itself, I see there is no provision for some of the Linux flavors, namely CentOS

These commands will not work:

# From https://github.com/mono/docker/blob/master/4.6.1.5/

RUN apt-get update && rm -rf /var/lib/apt/lists/*

RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF

RUN echo "deb http://download.mono-project.com/repo/debian wheezy/snapshots/4.6.1.5 main" > /etc/apt/sources.list.d/mono-xamarin.list \

&& apt-get update \

&& apt-get install -y binutils mono-complete ca-certificates-mono mono-vbnc nuget referenceassemblies-pcl \

&& apt-get install -y fsharp && rm -rf /var/lib/apt/lists/* /tmp/*

for the simple reason that the apt-get methods do not work in Centos. 

Consequently, one is forced to go back to installing the various components piece by piece. 

Next I will try to run in from Docker on the Google Cloud. Maybe I will have more luck.

If I succeed in installing it on Google Cloud I will publish the steps I took in a file so others can benefit.

Thanks

Serge

Probably a stupid question but, is there a guide on how to backtest/debug MY algorithm (python) on the desktop environment?  

I ended up putting my files in the lean folder and changing the config.json to look at my algorithm but is that the right way to do it? Seems kinda messy. I also got a LOT of depency errors and figured out that I had to import some extra stuff that I don't know why but needs to be imported at the start of the algorithm.

Haven't seen that in any tutorial, just something I picked up from other people's algorithms and figured I might need.

Also, Am I correct in the assumption that we can't backtest any algorithms with the full data that quantopian provides on it's web platform but only on data that we can download ourselves? I guess what I'm really asking is if I can use Visual Basic to better debug my code as I import it from Quantopian (LOTS OF ERRORS) and then use Quantconnect to backtest it. I don't care about backtesting on the desktop environment, I just don't know any other way to execute the code in a way that I can debug it.

Serge d'Adeskytry this one - Installing docker into Centos. With docker you shouldn't need to install anything (other than docker). Docker pulls down prebuilt images we've made & setup and runs the code in a virtual OS.

Hi Jared,



I don't know if you saw my last email. I'm kind of of stuck trying to

implement your solution. The docker file gives me errors, which I

documented on the file. I understand the kind of time pressure you may be

under and don't want to contribute to that But at the same time I think

there may be thousands of other former Quantopian users who are in the same

boat as I am and have even less knowledge and exprerience with Linux

servers, Python and C++ coding, etc.



If you are overwhelmed by this task, perhaps I can be of assistance? Let's

talk. I think there is a tremendous opportunity for you to capture a large

part of the 90000 plus users of Quantopian, but they will need a lot more

hand holding than you are currently providing. I have some ideas about that

that you may like that I would like to bounce off of you.



Can you give me a quick call. Shouldn't be more than 10-15 minutes of your

time.

Thanks

Serge d'Adesky

I'm far from a Linux expert, but I've found it's possible to manually provision, configure, build and deploy a Lean algorithm and a remote host in about 15 minutes. You could try following the install instructions on the mono homepage:

http://www.mono-project.com/download/#download-lin

I then also have a few tools to synchronize binaries and logs etc.

Docker is very well respected for microservices and continuous deploy but it's overkill for what I'm doing. I'm afraid my Linux knowledge doesn't stretch very far, but if you want to try a manual route I will try to help.

I should also say that host pricing means you won't make a significant cost saving (if any): I have to self-host Lean only because I run several modified builds.

Hi Jared, I appreciate your efforts to avoid vendor lock-in, however in this thread, https://groups.google.com/forum/#!topic/lean-engine/xeG-ZJMvx-4 , it was noted that Lean requires internet access and needs to connect to QuantConnect servers.  You advised that "We use the same charting HTML for the backend and frontend to save double-work. The charting packages are streamed to a web socket for display.".  

While I understand this might save you some double-work, does this not create a reliance on QuantConnect and creates a bit of a vendor lock-in?  I personally would like to be able to run Lean without having an internet connection or relying on QuantConnect's servers.  I ask that you consider adding this flexibility.

Thanks for the feedback Richard. Using LEAN does eliminate lock in but doesn't eliminate the need for you to do work =), if you'd like a custom display please feel free to use the open source one supplied or build your own. We don't charge for information; and break even on infrastructure; but we currently do charge for convience and saved effort/time. 

Hi,

I had a Raspberry sitting around without use, thus I decided to try Lean with it and I can say it works great. My algo has been running for a few hours right now and it uses between 1 and 1.7 CPUs and about 360MB of RAM, and a naked algo uses .5 cpus and 250MB of RAM. 

I wont be running it with a live account, since In my little experience with Raspberry, I dont find it to be reliable. Nevertheless, it seems a nice environment to run new versions of my algo with practice accounts, or scanning the market for opportunities.

Erik Bengtson awesome!! Please attach a photo =D

Richard, there are lots of options for drawing graphs from rollling data so you shouldn't have to use the QC client if you prefer not to. The QC engine already outputs data with all your trades and you can add extra log files with supplementary data. You will just need to set the environment to "backtesting" and you should be able to run offline.

Of course you can always go the caveman route like me and just dump your data into log.txt.

Here is the beast. I've taped it to avoid the power cable to disconnect

configuration: raspberry pi 3, samsung sd 16gb, Lean head version

quick setup:

• download the raspian lite image from raspberry site, and follow instructions (unmount sd card and copy image to sd card,...)
• now mount the sd card and create an empty file called ssh on the root folder
• now start raspberry with the sd card
• ssh to raspberry, and run
  • sudo apt-get update
  • sudo apt-get upgrade
  • sudo apt-get install mono-complete
  • copy compiled algorithm files to raspberry and run it with nohup
  • nohup mono algo.exe > nohup.out &

Hi.

Just browsed this forum.

Perhaps an aws image with everything already installed and with an update from repository script on build would make things easier for a lot of people.  Or even a vagrant image for local testing.  First day on Quantconnect - perhaps I'll try to contribute if this sort of thing does not already exist.

Hi, 

Is it possible not do partial sls opt out if I run lean on my own since it is prohibited in my country?