Hi Quants,

I'll be brief here.

I'd like to introduce you to vectorization in general and in Python.

What is vectorization?

In computer science, vectorization is applying the mathematics of vectors and their operations to arrays (people using Python know arrays as Lists) instead of using “for loops over arrays” to perform calculations.

Why using vectorization?

If you are using a computer from before 1995-ish, yes, you do not need to care too much about vectorization because your processor much likely does not really support true parallelism (multi-cores) and super-fast SIMD instructions on memory locations.

If you are using a computer from after 2000-ish, which I bet you do, then vectorization would make use of incredibly performant instructions in your processor AND would make use of those cores you are proud to have. So use true parallelism while doing calculations in your backtests/indicators. You can use it on your NVIDIA cards also, to use them for backtesting at just amazing blazing speeds, but I do not want to err there in this post.

Note that if you use QC web/cloud, you are using computers from after 2000……………………………… 😊. So vectorization is in scope for you.

How to use vectorization?

This is a college-level subject, even graduate-studies level on many aspects, so I will not fully teach how to use it in a forum post. Google is your friend. Numpy and Pandas, which are both included in the QC cloud environment so you can use them, are also very close friends here. They do have vectorization implemented for you. Look on Google. And, as you probably know, self.History() returns a Pandas DataFrame. So you can use vectorization right away while exploiting its results. Compared to using a for loop to iterate over the rows of the DataFrame, you'll gain 20-50-100-200-1000X…………

You could have a read of this:

and this:

As starters if you are more interested.

Pandas is incredibly vast and powerful. Even Numpy is orders of magnitude more powerful than most people make use of it. Think that vectorization is one of the foundations of data analytics. Now if backtesting and algo-trading are not data analytics…

SHOW ME CODE! SHOW ME COOOOOOOOOOOOOODE!!!!!!!!!!!!

Alright.

I did a very easy to understand comparison, to put things in perspective and eventually spark your interest.

Have a look at this:

  1. Timer unit: 1e-06 s
  3. Total time: 226.546 s
  4. File: test20.py
  5. Function: f1 at line 6
  7. Line #      Hits         Time  Per Hit   % Time  Line Contents
  8. ==============================================================
  9.      6                                           @profile
  10.      7                                           def f1():
  11.      8      1000  186045965.8 186046.0     82.1      l1 = random.sample(range(1, 1_000_000), 50_000)
  12.      9      1000       4287.7      4.3      0.0      _sum = 0
  13.     10  50001000   18733153.4      0.4      8.3      for item in l1:
  14.     11  50000000   21757529.3      0.4      9.6          _sum += item
  15.     12      1000       4978.6      5.0      0.0      return _sum / len(l1)
  17. Total time: 165.398 s
  18. File: test20.py
  19. Function: f2 at line 14
  21. Line #      Hits         Time  Per Hit   % Time  Line Contents
  22. ==============================================================
  23.     14                                           @profile
  24.     15                                           def f2():
  25.     16      1000  164171321.4 164171.3     99.3      l1 = random.sample(range(1, 1_000_000), 50_000)
  26.     17      1000    1223668.1   1223.7      0.7      _sum = sum(l1)
  27.     18      1000       3147.0      3.1      0.0      return _sum / len(l1)
  29. Total time: 168.975 s
  30. File: test20.py
  31. Function: f3 at line 20
  33. Line #      Hits         Time  Per Hit   % Time  Line Contents
  34. ==============================================================
  35.     20                                           @profile
  36.     21                                           def f3():
  37.     22      1000  168338808.0 168338.8     99.6      l1 = random.sample(range(1, 1_000_000), 50_000)
  38.     23      1000     632884.5    632.9      0.4      _sum = math.fsum(l1)
  39.     24      1000       3317.4      3.3      0.0      return _sum / len(l1)
  41. Total time: 268.41 s
  42. File: test20.py
  43. Function: f4 at line 26
  45. Line #      Hits         Time  Per Hit   % Time  Line Contents
  46. ==============================================================
  47.     26                                           @profile
  48.     27                                           def f4():
  49.     28      1000  167759822.9 167759.8     62.5      l1 = random.sample(range(1, 1_000_000), 50_000)
  50.     29      1000  100650675.6 100650.7     37.5      return statistics.mean(l1)
  52. Total time: 165.881 s
  53. File: test20.py
  54. Function: f5 at line 31
  56. Line #      Hits         Time  Per Hit   % Time  Line Contents
  57. ==============================================================
  58.     31                                           @profile
  59.     32                                           def f5():
  60.     33      1000  165213090.9 165213.1     99.6      l1 = random.sample(range(1, 1_000_000), 50_000)
  61.     34      1000     668202.6    668.2      0.4      return statistics.fmean(l1)
  63. Total time: 172.987 s
  64. File: test20.py
  65. Function: f6 at line 36
  67. Line #      Hits         Time  Per Hit   % Time  Line Contents
  68. ==============================================================
  69.     36                                           @profile
  70.     37                                           def f6():
  71.     38      1000  169385695.9 169385.7     97.9      l1 = random.sample(range(1, 1_000_000), 50_000)
  72.     39      1000    3595975.9   3596.0      2.1      _sum = numpy.sum(l1)
  73.     40      1000       4905.4      4.9      0.0      return _sum / len(l1)
  75. Total time: 0.380018 s
  76. File: test20.py
  77. Function: f7 at line 42
  79. Line #      Hits         Time  Per Hit   % Time  Line Contents
  80. ==============================================================
  81.     42                                           @profile
  82.     43                                           def f7():
  83.     44      1000     338150.1    338.2     89.0      l1 = numpy.random.randint(1_000_000, size=50_000)
  84.     45      1000      40276.7     40.3     10.6      _sum = numpy.sum(l1)
  85.     46      1000       1591.3      1.6      0.4      return _sum / len(l1)
+ Expand

I compute a mean/average in many ways. Is computing averages something you do in your algos? 😊

f1 : The sum is calculated with a “for loop”. Then the sum is divided by the number of items. Very classic huh?

f2 : I use the built-in sum() Python function.

f3 : I use the math.fsum() function.

f4 : I use statistics.mean() directly.

f5 : I use statistics.fmean().

f6 : I use numpy.sum() to compute the sum on a Python List.

f7 : I fully use vectorization implemented into Numpy.

Time needed to execute each function 1000x:

f1 = 226s

f2 = 165s

f3 = 168s

f4 = 268s

f5 = 165s

f6 = 172s

f7 = ……………………………. drum roll, 0.38s

Now, let's say I am telling you that using vectorization over Pandas DataFrame gives even bigger gains? We're talking about operations over vectors there but also, of course, over matrices. There are specialized CPU SIMD instructions that Pandas uses for operations over matrices for which gains over “for loops” are even bigger.

By now, I guess you understand that vectorization means you can crunch a lot more data for a lot more symbols in a lot less time.

Hope you enjoyed reading this.

Fred

Author

Fred Painchaud

March 2022