Recommended tier

Hi Ricardo,

Vast question.

1- Your code does matter. A slow algorithm is a slow algorithm.

2- Multiple calls to History. They create DataFrames from raw data which takes a while.

3- Using second bars. Since there are always many of them, when an algo processes seconds data, it has a lot to process. Even greater care must then be taken.

4- Going back and forth between Python and LEAN (.NET). There are two many patterns to try to avoid to list them here but in a nutshell, everytime your Python algorithm uses something in LEAN, there is a context switch. It is fast but still, it can accumulate.

5- Data processing in general is important, especially if it is intensive. Calculating the same thing over and over again on the same data, or on common subsets of data can lead to performance issues which can be eliminated.

Just a few things on top of my head.

Fred

Thanks for the insight, Fred Painchaud .

Question: what do you mean when you say “Going back and forth between Python and LEAN (.NET)” ?

My understanding is that by inheriting QCAlgorithm in our Python code, we are using LEAN. How does one go ‘back and forth’?

Can you give an example, (maybe with a code snippet) of this?

Hi .ekz.,

LEAN is implemented in C# on .NET. .NET uses the CLR (Common Language Runtime) to execute. Just like Java uses the Java Virtual Machine and Python uses the Python Runtime. In other words, CPUs do not execute C#, Java or Python directly. They execute assembly. Some programming languages are compiled to assembly, like C and C++, for instance, and others are interpreted (translated) to assembly by another program (runtime/virtual machine). That other program (runtime/virtual machine) is usually written in a language which is compiled to assembly but not always (you can implement, say, a Java Virtual Machine in Java) but at some point down the program stack, something must run assembly.

So, whenever your algo in Python uses or calls something in LEAN, that something gets executed in the CLR. When LEAN executes, 2 runtimes are loaded, if Python is used: 1) first, the CLR and second 2) the Python runtime. The CLR orchestrates the switch (I should write Python.NET in the CLR does) (we could call it a context switch) between the two. To start your Python strategy, the CLR gives way to the Python runtime, via Python.NET. Whenever your Python algo uses or calls something in LEAN, the Python runtime gives way to the CLR, via Python.NET, so that portion can be executed. Once that execution is over, the hand is given back to the Python runtime to continue its execution (any .NET results is marshalled (converted) into a Python object so the Python code can use it as if Python was executed). Etc.

That's the back and fourth between Python and LEAN.

Fred

Thanks for the reply Fred Painchaud  And thanks for confirming my understanding – this is what I thought was happening (I have a comp sci background, so this was my understanding). It's just the way you originally worded it that threw me off. 

It was worded as though we had an abundance of choice, not to go back and forth between Python and .Net, when in fact much of the back-and-forth happens under the hood as part of the engine – with code that we don't have access to.

With that said, i would LOVE to hear more about this : “There are too many patterns to try to avoid to list them here”.

If there are indeed some best practices to help us optimize performance by reducing context-switching, i think this would be a huge benefit to the community, and a worthwhile article to write. I could help. 

I'll ping you in the discord. 

Hi .ekz and Fred.

Just wanted to chip in.

A list of patterns to try to avoid seems like an fantastic idea. I have found myself using to many history calls, and slowing the algo down because of this. 

I dont know if I would have much to offer to the article written, but I would love to help 

Lucas

Hello,

Well, it was well worded then. 😊 The abundance of choice we have not going back and fourth between Python and .NET is infinite. Countably infinite, but infinite. Because there are a countably infinite ways to write algos that stay on the Python side. Yes, in a way, you do not have access to the .NET code (well, in a way, because it is open source so you do have access to it but it is something different here) but you can predict where and when your Python code will go back and fourth and thus what to avoid. The catch 22 is that to avoid going back and fourth, you need to avoid “calling LEAN”/"using LEAN" and thus you need to implement the aspects you do not call yourself.

Calling History is a good example to discuss. First, all calls to History are not equal. The time spent in History depends on a) the requested period of time, b) the requested resolution, c) the number of requested symbols and d) the type of symbols. Calling History to get 365 days of Resolution.Daily for one symbol is not a problem. Calling History to get 365 days of Resolution.Second is. Even 30 days of Resolution.Second is. Calling History on seconds for crypto/forex is going to take much longer than equity because of the longer trading hours. Etc.

Second, avoiding History is “difficult” because it then means you have your own data feed. In a backtest, it is ok because you artificially avoid History by beginning your backtest later in the past and use, for instance, a rolling window to get the data as it comes. But once you deploy live, you cannot start your algo in the past. You will need History to pump past data into it so it is ready now - not in X days. Well, of course, you could avoid History even live, if you flag your algo not to trade until your rolling windows are ready. That's the live version of “starting your backtest later in the past” which is “starting to trade later in the future”.

Third, History can take a while on some requests but is embarrassingly parallelizable (https://en.wikipedia.org/wiki/Embarrassingly_parallel). For C# algo programmers, it is business as usual. For Python, it is not so easy however to code real parallel algos since the CPython implementation (the most common one - from python.org - and the one used by LEAN/QC) has been implemented not to support real parallelism on multiple CPU cores. Read “it uses something they called the GIL - Global Interpreter Lock”. So a CPython runtime cannot run code on multiple cores - just one. You need multiple CPython runtimes to do so (read “use multiprocessing”) or you need to use libraries that do so. A very good example is Ray (https://www.ray.io/) - it is a very good example because it does not only apply to parallelization but also to ML, crunching data, etc so it is in line with algo trading. But there are many others, as you can imagine. The point in parallelizing calls to History is not to make a 2-hour request take 1 second. It will take around 30 minutes on 4 cores. It is to make LEAN execute something else while also serving your History, the same to your algo - to make History asynchronous, to use a single word. I have been testing Ray locally with my own data loaded through History with great results. Ray gives you the power not only to parallelize on a single machine but also on a cluster (think as if you had a few backtesting nodes or even live nodes - and you can balance the entire workload of your algos on them). Pumping data asynchronously, as you know, then means your algo says “ok, pump that data on the side, prime everything up with it, and once ready, start to trade those additional symbols - I will continue my current trading while you do so".

So.

The list of best practices to avoid context switching is really a book (in my mind at least). Something like “LEAN for Python algo-traders". I don't have that kind of time, sorry. One metric I can see useful to add to LEAN is “the number of context switches you had during your algo execution”. That could be reported with all other stats in the backtest results. Since QC forked Python.NET to implement some changes in it adapting to LEAN needs, I would bet it would not be very hard to add the “context switch counter" in there. The time spent in context switching can be more tricky to get right however. But an approximation is probably also do-able with a reasonable amount of effort. That could also be a second useful metric.

But all that is just one aspect of performance. My first point above is really the most important one (a slow algo is a slow algo). If you have, say, a dozen spots where complexity is not optimal, where it is linear instead of logarithmic, or quadratic where it could be nlogn, etc, then when you execute those spots thousands and thousands of times, the slow down is very substantial (read thousands and thousands of times slower - literally). So when optimizing, after complexity is right, then you go through the long list of other tweaks - with the help of profiling.

Please get me right here. I am not saying “you are bad”. My thesis has always been my first sentence:" Vast question. ". That's what I tried to demonstrate.

Fred