Share: Kalman Filter Crossovers for Crypto and ‘Smart’ RollingWindows

ekz,

Great codding, congratulation.

Yesterday we published "Trading Kalman Filter" - the very simple version with only one indicator. and only one parameter which was not optimized yet.

Compounding Annual Return - 237.024%Drawdown - 38.500%The ratio of the annual return to the maximum drawdown is 6.16.

Compare to yours, with 3 indicator and 3 parameterCompounding Annual Return - 281.711%Drawdown - 50.300%The ratio of the annual return to the maximum drawdown is 5.60 and less than in our simple version.

In our practice, to compare algorithms, we use the ratio of the annual return to the maximum drawdown as the first metric, because we can normalize the return to the accepted maximum drawdown by simply reducing the exposure. 

We usually start creating trading algorithm with the simplest version and then adding indicators and parameters one at the time only if they improve risk metrics of our choice. 

Best Regards,

Vladimir 

Hi ekz,

Very cool algo, good job. The returns are amazing but the downside risk very heavy, why don't you sacrafice some of that return to hedge your position?

Hi Vladimir! Thanks for the reply and the kind words.  CAR/DD ratio is a good metric to use for sure. I factor it into my algo performance criteria as well and I agree this is not tradable as is. As you rightly pointed out, for some assets using Laguerre and EMA crossing Kalman is less performant than your original version that just looks at price crossing Kalman. The fun is only just beginning, to research even better performance across the board.

On a related note: do you know how to modify the gain of the Kalman filter in Pykalman? I'd like to attempt Kalman-vs-Kalman crossovers, with more responsive vs less responsive, but to do so I need to modify the gain and I dont know how to do so with this library.

Mak K : Thanks a lot and, agreed! What are the different ways you would consider hedging positions in such a system? Would love to hear thoughts. 

Here’s my updated version, with Ret/DD at 9.69

• CAR 299.561%
• Drawdown : 30.90%
   

Changes:

1. Noticed Vladimir’s new version uses the ‘low’ of the bar for the Kalman calculation. I've now modified the Kalman indicator so it can now take any of the selectors (open, close, low, median, average, weighted, etc). Low performs the best.
2. I tried a few different variants of EMAs and Laguerre filter, as well as Vladimirs version of price above or below Kalman, but price crossing above/below seems to work the best.
    

Note: I am already doing this but noticed others might not be: setting an explicit brokerage model (instead of the default) also impacts performance. I'm using Bitfinex, where I plan to go live, and perf is significantly better. 
 

.ekz.

-->  I'd like to attempt Kalman-vs-Kalman crossovers

There are thousands of ways to apply the Kalman filter to trading algorithms, from estimating the next bar's prices and the standard deviation of prices, to generating signals based on forecast errors and dynamically calculating hedge ratios. 

A couple of years ago, we created over a hundred Kalman Filters applications on Quantopian that have not yet been ported to QuantConnect.

We'll see if the one you are looking for is there.

Here is slightly modified “Simone Pantaleoni btcusd kalman-mom strategy”

Interesting share indeed. I have ran this with ETHUSD and the results are attached. Looks like it is giving better returns. (.ekz latest version from above)

Would be nice if we can test it out with leveraged crypto futures or crypto options (maybe use it as hedging of some sort).  

ekz.

-> Vladimir : 

for the Kalman filter, how come you have such a long warmup period? 

IE: Why 5 * Period? What is the significance of the constant '5'?

From digital signal processing point, Kalman filter is adaptive infinite response filter (IIR).

Current value of any IIR substantially depends of starting value of time series.

In our practice we usually use 5 times more data points to warn up any IIR to mature.

Great to see the discussion picking up!

spacetime: Awesome that you are trying this on other cryptos. That's the idea. I’ve also tried it on ETH and SOL, and I feel the drawdown can be mitigated more.

Vladimir : Good insights.  It makes sense then, that the period will influence the response filter performance. Also, thanks for the version that factors in momentum. I will play around with it, but I admit i am wary of the introduction of multiple parameters that are related to time periods (eg:lookbacks).  From what I’ve learned about Kalman, a core part of its value is the fact that it adapts effectively, with much less dependency on lookback compared to, say, moving averages.On a related note, Vladimir, your current entry condition (price is above kalman) unfortunately leads to overtrading, because even though you are already invested it will sell and re-buy. This leads to about 443 trades. Adding the condition if (not self.Portfolio.Invested ) reduces this to 81 trades with slightly better performance, and a savings on transaction fees.

Trading a Crypto portfolio:
For anyone else looking to tinker with this: something to keep in mind as a ‘guiding principle’: Ideally we should able to trade this system with a portfolio of cryptos —imagine daily scanning for whichever security meets the entry criteria, and when we have several,  holding them with equal weight (or some factor-based weight).  In that case, we’d want a generalized system that is NOT fit to one particular security. This is why I’m hesitant to add new time based parameters because they are likely to have different requirements (e.g. the favorable momentum period of SOL will be different from that of ETH, ADA, or BTC).

Updated Exits With Dual Kalmans Crossing: To reduce drawdown, I investigated other exits (other than price crossing below the Kalman --this happens quite late), and was able to test a responsive Kalman crossing below a less-responsive Kalman. I was able to increase the responsiveness by updating the transition_covariance property (higher value makes it more responsive and follow price more closely). It seems to work alright, squeezing out a better CAR/DD for BTC (298% / 29% = 10.3 ), but not so well for ETH and SOL --also, the optimized values are quite different for different securities which, as mentioned,  is not ideal… 

Also, because this exit signal tends to fire early sometimes, I updated the entry signal to match vladimirs, so we jump back in the trades whenever price is above the original Kalman (previously i was only entering when price *crosses* over the Kalman.

All that said, it's still worth sharing. See the attached backtest. 

A small observation.

The entry and exit method 3 is the only one being executed. Methods 1, 2, and 4 do not happen.

Method 1 and 2 happen after method 3 has triggered a trade. While method 4 is never reached nor executed.

Hi Guy Fleury , great to see you chiming in!

You are correct.  I parameterized entries and exits to easily test different mechanisms. Only one entry (and only one exit) method can be used at a time.

As of my last version, here are what the methods correspond to: 

Entries:

• Method 1: EMA crossed above Kalman
• Method 2: Laguerre crossed above Kalman
• Method 3: Price crossed above Kalman
• Method 4: Price is above Kalman"
• Method 5: Price crossed above Kalman OR (price already above Kalman & rising & Fast Kalman > slow Kalman)

Exit

• Method 1: EMA crossed under Kalman
• Method 2: Laguerre crossed under Kalman
• Method 3: Price crossed under Kalman
• Method 4: Price is below Kalman
• Method 5: Fast Kalman Fast crossed under Slow Kalman

Right now, My preferred Entry/Exit pair is Entry #3 + Exit #3, as they seem to be the most consistent across securities, and require fewest total # of parameters. 

Hope this helps. 

.ekz.

On your latest version above, all entries are done on method 4 while all exits are done on method 5.

Is there a reason why most of your trades occur at 20:00:00 and some at 19:00:00?

Guy Fleury  Again you are correct.  

In my last attached backtest, I was testing entry method 4 and exit method 5. I described this in the body of the message/post. Ultimately i still believe entry #3 and exit #3 are more suited for trading multiple securities. 

Not sure why some of the trades occur at 20:00 and some at 19:00.  It might have to do with the way Lean consolidates the data, perhaps. Good catch, though. It might be worth investigating.   

In case it's not obvious to others: i am using externalized parameters, and you can see them in the left pane of the IDE. 

.ekz.

In your strategy, you have 5 trade triggers for entries and 5 for exits. The order in which they are placed can determine which will be executed first since only the first one reached of each group can be executed at any one time, as you said. You can also force which one will be executed which is what is being done here.

Theoretically, over the same lookback period, the Kalman filter is more responsive to price movements than the EMA signal which is more responsive than the Laguerre signal which in turn is more responsive than the SMA signal. If you test for the price to cross the fast Kalman filter first, the other cross-overs will not even be tested, and, therefore totally redundant. Price precedes them all. 

In Vladimir's version, method 2 (Laguerre crosses Kalman) is the more productive method. The same goes for your version. However, your method 5 does outperform them all.

In this strategy using the Kalman filter as a cross-over system you simply have a variation of the old EMA cross-over trading strategy. It becomes something like finding the best lookback period for an EMA cross-over on some past market data. This might also raise a curve-fitting flag. That lookback period might not hold or be the best going forward.

Bitcoin is, let's say, volatile. It can swing 20% in a day (it has done so a few times). Any fixed lookback period could be costly. Maybe something even more adaptive might be needed, but then again, you would be back with the EMA cross-over thing. Almost suggesting that maybe minute data should be investigated next.

The strategy has no fundamental data support. The reason is simple: none is available. There is no formula that can say this is how much Bitcoin is worth. We cannot say either that Bitcoin is over-priced or under-priced at any given time. Nor can we forecast a future value within any confidence interval. You can only rely on lagging technical indicators for trading decisions as was done in your trading script. On the other hand, we could just say it is going up until it doesn't which would put you back at the right edge of the chart.

The big question is: does the price preannounce where it is going or do we just trail behind to wherever it wants to go?

I found the CAGR to be impressive. Good work.

Hi .ekz.,

You could also investigate variable lookback periods based on volatility (such as ATR). When volatility is higher, lookback is lower. And vice versa. It gives you faster response when needed (in big swings which normally come with greater volatility) and better statistical significance when needed (sideways market - flatter volatility).

Such as done within Dynamic Momentum Index, for instance.

Fred

Guy and Fred, thanks for the feedback!

Guy Fleury  (on evaluating the system):
Love the breakdown, and appreciate the detail you put in your response. You raise a very good point, too, and I landed at the same conclusion (that Vladimir highlighted to his credit):  Price is the most responsive to price (ha), and so using this as a crossover signal will beat them all. 
Also, you said it correctly about the Kalman crossovers, we definitely run into the problem again, with subjective parameters. because not all securities created equal... if only there was some way to normalize the lookback/parameterization... :)

Fred Painchaud  (on dynamic lookbacks):
... which brings me to your point, Fred. This sounds like a good idea! Adjusting the responsiveness based on volatility, similar to how the Kaufman adaptive moving average adapts based on market efficiency (different from volatility, but comparable). With some adjustments, we may be able to get the appropriate responsiveness for the corresponding market voaltility (or market noise), regardless of the security. If trading on the daily, perhaps we will want to look at the volatility/noise on a lower time frame --perhaps the 4 hour. I will think about this and see how best to apply the logic. If you have any examples of this in practice, or any additional guidance, please share!

Everyone (on 'Strategy' vs Testbed):
Everyone, one thing to note: the backtest i've shared isnt exactly a 'strategy' as is. It's more of a strategy testbed. This is why you see multiple interchangeable entry and exit methods in there --it's easier to test and compare them against each other (using the optimizer). Typically, once I've tested and determined the ideal entry/exit method, i remove the others.  I will probably remove the non-preferred ones soon, but I may want to try one or two more things first.

.ekz.

Here are the test results of both versions (yours and Vladimir's) when forcing the program to only execute each method in turn.

The first 4 methods gave identical results. The second method was the most productive (Laguerre crossed above/below Kalman). This is a slightly delayed crossover when compared to using price. It was like shifting a position slightly forward in time. 

As can be observed, your method 5 did even better.

Hey .ekz.,

Glad you liked the idea. It's really as old as mathematics.

An easy way to dynamically compute periods is to take one measure of volatility and divide a chosen "mid-period", adequate for the metric at hands to “dynamize”, by it. When your measure of volatility is greater than 1 (rising), your “mid-period” decreases. And vice-versa. Like pretty much every metric on y-half-bounded series like price action and volume, your measure of volatility will in theory be able to go as down as 0 and, however, won't have, in theory again, an upper limit. So you'll need to clip the resulting ratio ("mid-period"/volatility), or else, you'll end up with potentially a division by zero (and very very high resulting ratios when volatility is close to 0) and then also potentially very very low resulting ratios when volatility gets very high (the limit calculus being 0).

So, in a nutshell:

Volatility:

• Can be any of the multiple metrics already known.
• Say, you can base on ATR. For instance, atr(5)/sma(atr(5)). Here a period of 5 is an example. You certainly do not want to calculate volatility on a too long period since you want it to react fast.
• Or you can base on close. For instance, stdev(close, 5)/sma(stdev(close, 5)).
• I mean, there are multiple multiple ways to capture volatility. It's rate of change of signals. It's everywhere in engineering/math books.

Mid-period rebase:

• You look at the metric you want to dynamically parametrize, your TF, and think about a “normal”, “good”, “usual” period for it. Something that is not short neither long…
• You divide it by your volatility ratio above.
• You clip it both min and max. So here, you think about the shortest period you want to use. And you think about the longest period you want to use.
• For instance, your mid-period could be 14. 14 daily bars is typical. If you are at the minute res, 60 could be typical. ETC
• So you would divide say 14 by the ratio above, floor it and make it say 3 if lower than 3 and say 52 if higher than 52. Again, numbers are examples. 3 can be much too low as a period for the dynamic metric and 52 way too high…

That's it. Well, that's one easy easy way.

Many normalize (read, rescale between 0 and 1 here - as normalization means quite a few things) volatility, usually with artificial stochastics based on running minimums and maximums (stochastics don't apply to samples. but anyway), and then multiply by the mid-period…

Electrical engineers would yet tackle the problem differently. Etc.

Enjoy!

Fred