Share: Kalman Filter Crossovers for Crypto and ‘Smart’ RollingWindows

Guy Fleury : Thanks! To be fair, Vladimir's version is the winning one: his original code uses price above/below kalman, and doesn't have the multiple methods. I built on his code and added multiple methods, including the Laguerre-Kalman crossover. Thanks for sharing these screenshots.  I also ran comparisons of them all, using the Optimizer (and incrementing method # to 1,2,3,4,etc).  Curious, which assets are you trading in these backtests, and over what time period?  Please attach the winning backtest when you have a moment. 

Fred Painchaud : Love this. Very informative and will help as I try this out. Not surprisingly, the KAMA indicators uses something similar.  It uses the Kaufman efficiency ratio for the noise measurement, and also takes a floor period and ceiling period. It uses market noise though, and I think volatility** might be a better modifier for what we are trying to do here. I will share my results when done! 

__________________

**Something I learned recently: Market Volatility ≠ Market noise. Sharing this link for the sake of others. 

Yup. Short addition. TL;DR. Noise is specific to a signal/system. It's unwanted signal (disruptions) generally closely around main/wanted signal. Well, if noise is 1) frequent and 2) powerful wrt main, then, you have a problem getting ("hearing" so to speak) your main signal, you loose it in the noise, you can't filter it out. Enter signal-to-noise ratio. In trading terms, noise is retracements and big money checks, for instance. Volatility is another thing indeed. It's a rate, just like acceleration (rate of velocity change over time). It's the rate of price change over time. If you use noise to dynamize a lookback, you might end up biting to big money checks. Except maybe if you weight noise with volume. Volume will most likely attenuate noise created by checks enough so you don't signal trades. But I don't think I would bother. Well, my current opinion…

.ekz.

Here is my solution how to calculate volatility-adaptive self.kalPeriod.

You've probably seen it in my thread Intersection of ROC comparison using OUT_DAY approach.

        self.history = self.History(self.ticker, self.VolatilityPeriod, Resolution.Daily) 

        vola = self.history[self.ticker].pct_change().std() * np.sqrt(252)       

        self.kalPeriod  = int ((1.0 - vola) * self.kalBasePeriod) 

I'm sure you know how to find the best combination of self.kalBasePeriod and self.VolatilityPeriod.

These are good analogies, especially how volatility is a rate. Simple and true. 

Not sure I am following what you meant by this line though:  If you use noise to dynamize a lookback, you might end up biting to big money checks.  “Biting to big money checks” … is this a good thing? 

Also, weighting noise with volume sounds interesting.  Would the goal be to determine whether the noise is ‘real’ noise? If there is low volume vs high volume? 

So as to keep this thread on-topic, we can continue this sidebar topic via chat. Are you on the Quantconnect slack or discord?  I'm in both, with the same name: ‘ekz’.

Vladimir: Thanks for this. I will give this a try ASAP.

Very much appreciated!

.ekz.

The only changes made to your version were the start and end dates.

self.SetStartDate(2015, 1, 1) 
self.SetEndDate(2021, 11, 9) 

.ekz.  valdimir 

I'm trying to build a black swan hedge, running in parallel with a kind of In-Out strategy.

The idea ist to profit from the VIX spikes. If one buy very far out of the money call, like 100 Vix, they start to spike like crazy, 200-400 times the purchase price!!, please see the the link for the code.

To decide on the liquidation, I used a classic MACD, as well a MACD with a Kalman filter, like the examples in this post.

Everything much too slow response, some suggestions? Thankx

.ekz. First of all - Thanks for the contribution! really nice algorithm structure and I really liked the SmartRollingWindow class.

I've been playing around with taking this structure and trying to apply other indicators to it, most of them didn't come close to the performance seen here, but when I applied a rather simple EMA crossover with ETH I actually achieved a nice reduction in drawdown while maintaining the PSR and returns,

I wonder if I'm missing something in my code, or is this a valid crossover approach using simpler indicators

Nitay Rabinovich: That looks great! Sometimes the simplest things work best :) It's been long suggested that 'old-school' trading strategies work great for crypto since it's a fairly nascent and inefficient market, so I'm not surprised the simple EMAs work well! Might be a good idea to try the logic in a crypto universe and see how it works. Vladimir & Fred Painchaud: I switched into holiday mode so I've been quiet, but I played around with the volatility-adjusted period in isolation and didn't see much success, unfortunately. I saw better results across assets using ATR based dynamic stops that I'm more familiar with (a multiplier of the recent ATR).  Will share a universe crypto trend follower soon. I might just use a EMA version like the one Nitay Rabinovich shared. 

Carsten: I played around with black swan hedges using SPY puts. Haven't tried VIX hedges, but I've heard they're generally more successful. Good luck!     

.ekz. - Happy holidays! 

I actually wanted to test if this EMA approach is only fitted to ETHUSD, so I modified the code to fit multiple assets, I manually set BNB, ETH, and SOL. And I set the benchmark to BTC (which is… questionable I guess?) 

Maybe it would perform even better with dynamic universe selection, but overall seems like the concept works well with multiple assets as well, I did see some issues with orders so I think placing orders needs fixing.

Nitay Rabinovich - That is awesome! Simple, very low drawdown and high PSR. IMHO using BTC as benchmark makes total sense. I have also tried to test it with other MA types, but EMA produced the best results so far. As for using a dynamic universe selection - that is great, the question is - what parameters to use in order to pick a specific crypto (volume? other factors?). I also wonder if similar approach can be used for short positions - for e.g. when slow is above fast and price is below both fast and slow.

Anton Kiselev - I'm really not that experienced with universe selection in general, but if I'll find the time I'll try to research some approaches to filter probably by volume, volatility, and liquidity (probably via https://www.bitcoinmarketjournal.com/token-velocity/). and there's also the question of the rebalancing period.

Regarding short-selling - it's an interesting idea that's definitely worth checking! 

Good stuff Nitay Rabinovich, applying it to a basket of cryptos. I have some crypto-universe code that may be helpful --it includes volume thresholds, rebalancing logic, and a few other handy things. Will share after cleaning up a bit.

In the meantime, one thing i recommend you do for your current system, is running it with Minute resolution --doing this will give you higher precision, using bid and ask prices, and the results will be more reflective of what to expect in "real life".

.ekz. - So I tried a couple of things - 

1. SImply changing the resolution for all to minute data, including the consolidation handler and rolling window updating - caused way too many trades, pretty much killed the algorithm with excessive trading.

2. Tried to mitigate with consolidating on daily resolution but using minute data for the cryptos and indicators - also reduced performance massively. (tried with several different periods to see their effect, but below you can see the best case)

Still has excessive trading…

3. Lastly I changed the crypto's resolution to minute, yet kept both the indicators and the consolidation handler on a Daily resolution. And that… well created this amazing backtest. 

So the question is - what's to expect in “real life" - I know for a fact I won't run an algorithm trading crypto per minute, so 1 is out of the question, but is using daily resolution for indicators and consolidation handler considered “painting a pretty picture" that only works in backtests? No matter the answer, it is a pretty backtest :) 

Nitay Rabinovich #3 is the way to go, and I'm glad it looks good! 

Over the weekend I'll take a closer look to see if any bias has been introduced that may invalidate these results. I will also share the additional universe code.

Good stuff!

Hi Nitay Rabinovich, I did a more thorough walk through of your last backtest and noticed/fixed something:

TLDR: 
Currently, when we look at our rolling windows for exit/entry signals, we were looking at old data (yesterday's data). Addressed this by changing where we update the rolling windows. Overall performance is impacted negatively. 

Details: 

1. Currently, Every day, just before midnight, we update the rolling windows used for entry/exit. This includes the price rolling window. They are updated at 23:58:00, from a call to self.OnEndOfDay()
2. Every day at 7pm  (19:00), the LEAN engine updates the ema indicators (this is when LEAN’s daily consolidator consolidates the daily bar )
3. Also at 7pm,  the code is checking for exits/entries, in the method self.OnConsolidatedBarClose
4. At this point, it is using the old values in the rolling windows from midnight the previous day — we are entering/exiting 19 hours after the signal fired

In the attached backtest, I made the code change to address this. Now we are updating the rolling windows at the right time, right after indicators are updated. So at 7pm (19:00), the sequence of events is now:

1. Indicators Updated
2. Rolling windows are updated
3. Check if we should exit/enter

Note: 
I changed the name of the method from “UpdateIndicators” to “UpdateAssetWindows” to be more accurately reflect what the function is actually doing. Also, OnEndOfDay is now only used for plotting charts..

Code diffs: 
https://www.diffchecker.com/O6AKaeJ9

Take-Away:
The results arent as favorable (less profit, higher drawdown), but i think this is a necessary change for more predictable behaviour. That is, unless we can rationalize why it is better to use yesterdays prices to make decisions today. Perhaps that can be part of the strategy, but it does seem that it would be arbitrary at best. 

FYI: I started a new thread for EMA crossovers, so we can keep this thread focused on Kalman filters

Nitay Rabinovich : let's continue the EMA conversation there, pls.

Vladimir : I have a follow up question up on our earlier topic, calculating volatility-adaptive lookback period. Specifically around this code:

self.history = self.History(self.ticker, self.VolatilityPeriod, Resolution.Daily) 
vola = self.history[self.ticker].pct_change().std() * np.sqrt(252)       
self.kalPeriod  = int ((1.0 - vola) * self.kalBasePeriod) 

My question: How do you determine the best combination of self.kalBasePeriod and self.VolatilityPeriod? 

Hey! I've been following the development of this strategy and have been trying to learn from all the insight that all of you have shared, super interesting!

Just a thought. From what I understand, this is a long only strategy. For the sake of robustness, would it make sense to test the strategy's performance trading in both directions? With the cycles that any asset goes through, would we be inadvertently overoptimizing the strategy by only testing a long only strategy when the crypto market has seen an incredible bull run?

Hi Ariel Nechemia, thanks for the note, and sorry for the late response. 

So, going long in a bull market isn't over-optimization, it is arguably conventional wisdom. IE: it's what you do. 

Similarly, systems that trade short only strategies often have a regime filter to make sure the market is in a downtrend before taking any positions.  We are doing the same here --we are using these indicators to detect bullish price action, and taking long positions accordingly.

You can certainly make this bidirectional (long/short), and introduce additional conditions with the inverse logic (swap ‘<’ for ‘>’). This would effectively introduce bearish regime filters, during which you would go short.

Hope this helps!