Realistic expectations in algo trading

Welcome!

You're asking very good questions, but if you asked 100 people, you'd get 100 different answers. It depends on everyone's personal experiences. It also heavily depends on what you consider a "profitable" algo.

E.g. you write an equity strategy that outperforms the S&P by 50 bps per year. In a market downturn, the S&P drops by 40% and yours drops by only 39.5%. Some would call a consistent outperformance like this one "profitable" and others would not.

If you tell us a bit more about your own expectations and what kind of assets you'd like to trade, it's easier to say whether they are reasonable or not.

There's not one correct answer to any of the questions you've posed.

Personally, I'd advise you to not think about expectations but rather to think about what you want to learn from this experience, and let that guide you. There's certainly room for retail traders to make money and you're embarking on the journey at time when technology and data is becoming more easily available - QuantConnect is a perfect example of that.

I think the most important thing to realize is that 99% of algorithms have bias which makes the backtests extremely unreliable. I can come up with a backtest for an algo that will give you 10000% returns in 10 years but it won't work in reality. Most of my effort writing algorithms has been removing bias and fixing errors. It certainly is much harder to execute profitably in real life than on paper. If you are shooting for the moon you will likely fail, by that I mean attemping any sort of intra-day algorithm, "HFT", or trying to translate some super advanced academic paper into a profitable algorithm the chance of success is nearly 0% so don't waste your time.

That being said, if you are looking to reduce downside risk and maybe bring some intelligent rebalancing to your long term buy and hold portfolio there is definately room for improvement. Most retail investors are not allocating their retirement savings optimally, maybe rebalancing once a year or buying/selling due to emotional reactions like panic or greed. You can make an algorithm that "beats the herd" so to speak as long as you can weed out biases. As much as you can, try to minimize downside risk rather than chasing profits.

Hi Douglas,

Thanks for the quick reply!

So let me explain a bit more about my perspective. 

I'm a (non-professional) trader who is mainly active in forex, but also in stocks and other assets like bitcoin, oil, gold, silver, etc... 

I'm trading purely based on TA. During my almost 1 year ongoing learning journey (short but very intensive) I encountered so many statements from traders in books or on social media about strategies, indicators and all kind of patterns that I (the engineer that I am) always felt sceptic whether they are really statistically evident or just bogus.

Since I find it very difficult, mistake prone and boring to manually backtest those strategies I started to code some of those (candlestick patterns, divergences, etc.) in Tradingview to see if there is something. 

Unfortunately TV uses pinescript which is very limited so I searched for a way to do that in python. Later I came across Quantopian and eventually Quantconnect where I cannot only backtest but also trade the algos live, which is very exciting, so here I am!

There are 3 major reasons why I'm interested in algo trading:

 - Evidence based strategies to increase confidence.

 - Fits better with my lifestyle (limited screen time due to family, day job).

 - Less emotional involvement (although not expecting zero emotions obviously).

Profitable to me is a strategy that consistently increases my capital with a percentage that is comfortably compensating for the effort that I put into it. And all of this with an acceptable drawdown (Sharpe ratio). I think I’m not the kind of trader who can sustain a win rate of 20% with a few homeruns to make the year. I try to have realistic goals, but I'm definitely aiming to outperform the S&P for example. 

When you're used to forex long or short doesn't matter as long as you get the direction right. I’m not interested in HFT. I like the medium to long term timeframes from 4H to 1W, but I wouldn’t totally dismiss LTF like 1H or even 15m if feasible.

I think I have understood the very basic concepts of algo trading including the most common pitfalls like overfitting, look ahead bias, in sample and out of sample testing, transaction costs, etc., although I have obviously no experience yet how to conduct all this properly into code.

I’m sure I forgot some things to mention, but I hope now you have a better understanding of my background!

Cheers!

Thom

Hi Lukel,

I share your points, there are many stupid algo strategies on TV claiming absurdly high profit percetages, but when you look inside the code you will see that they suffer from look ahead bias for instance.

And no, it's not about creating an algo that makes you rich in 3 trades. The fundamental rules of trading like proper risk and money management will also apply in algo trading. But an algo that can trade profitably some 5 to 10 swing trades within a week from the 4H chart would be very nice! :-)

Hey,

Couple of thoughts when reading your post:

• Sharpe ratio has nothing to do with drawdown, although often you see a relationship between drawdown and volatility.
• You mention being familiar with some important aspects of systematic trading. This is great, and it's important to remember these when you eventually come across a strategy that looks promising - never forget to test it rigorously and apply the same qualitative and quantitative checks to each and every strategy you develop.
• You sound like you are an "absolute return" kind of investor - looking for consistent (albeit sometimes small) gains regardless of market conditions. As such, I would forget about comparing your strategy to the S&P or any other single instrument.
• Use Sharpe ratio to compare different strategies as well as your drawdown or CVaR (closely related). Figure out what kind of leverage you can get on the instruments you trade, but also make sure you fully and completely understand how your broker calculates margin requirements and when/why/how it will close you out of a trade.
• Start out with the expectation that slightly more than half your trades will be loss-making - anything better than this is good. Focus on making your winners go further than your losers.

Yes, you're right of course with the Sharpe Ratio and the comment about the S&P, I don't plan to use the S&P as a benchmark, wouldn't make too much sense when trading forex etc anyways. What I meant is that my goal would be to have a higher return than 7 or 8% per year like an index like e.g. an S&P would have. And yes, a win rate of 50% would be already acceptable together with an R>2,  I would prefer something like R>3.

Coming back shortly to my original question regarding the expectation. From your experience is it likely to find a "winning" system with simple strategies like comparing some candlestick or indicator relations? Or would you say that it will have to be way more complex to achieve the mentioned metrics?

Thanks for your comments Douglas!

In my view, your goal should not be in terms of % return per year as it's an arbitrary number. You should include a vol target in such a goal in order to measure what risk you're willing to take to return such gains. A 1% per year strategy can be "better" than a 10% per year strategy if the former has lower volatility (relative to its return) than the latter. This focus on Sharpe ratio is not just fund managers wanting to blindly compete on a number - there's little benefit in comparing two strategies based purely on their returns. High Sharpe ratios also often let you leverage more (high sharpe -> lower vol -> lower drawdowns -> higher leverage without being margin called).

Regarding your expectations: everything is possible. From my experience, there's a trade-off between difficulty of development/research and difficulty finding profitability. Strategies that are easier to develop can be harder to turn profitable, and vice versa. Although this is definitely not always the case - there's plenty of examples to show that simpler strategies outpeform more complex ones out of sample (see trend-following and diversification).

In my view, you should neither listen to someone who says "this strategy is impossible to make money from" nor to someone who says "this strategy is guaranteed to make you money". The truth is somewhere in between and depends on your own implementation, dilligence and commitment.

If I were you, I would have no expectations whatsoever and instead start with the techniques you know, building very simple strategies to start with. Analyse how each one is performing but also why it's performing the way it does. Learn from each one and successively expand your toolkit.

Some pointers in no particular order (from my own experience, some may disagree):

• Watch out for trading costs - trading more often should improve your Sharpe but will incur greater trading costs
• Diversify, diversify, diversify - a bunch of low-Sharpe strategies put together can increase your Sharpe greatly
• Keep your number of parameters low - easy way to reduce potential overfitting
• Be curious - try lots of different approaches but be prepared to drop them quickly if they don't work
• Verify, verify, verify - get accustomed to debugging your code a lot. It's very easy to get things wrong, and it's better to catch bugs sooner rather than later. Even if a line of code seems bulletproof, get accustomed to checking that its value is what you expect it to be. Use Excel or other software to verify the behaviour of your algorithm
• Watch out for drawdowns - high returns are great, but drawdowns will wipe you out. You might even want to look at the Sterling ratio (return over drawdown) rather than the Sharpe ratio, or at least have a max upper limit for your drawdown

Hey Douglas,

thank you very much for your extended reply, it's highly appreciated!

I think we are on the same page with regards to taking care of the drawdown. I guess I didn't make it clear enough in my post above, but it's definitely a metric that I consider important. As I said, I want to avoid strategies with too lareg drawdowns due to the psychological burden.

Trading costs for example is a point that I already try to get my head around on how to incorporate this in my code. I try to play around with "CustomFeeModel" and "CustomSlippageModel", but there's still some way to go.

At the moment I try to work myself through the basics of coding in QuantConnect a basic forex trading algo framework that can do all the standard features like setting up the account (symbol, resolution, brokerage, max leverage, ...) and other elements like order management, portfolio management, rollingwindows for indicators, debugging, postprocessing and validating, etc. I read a lot in the documentation and in the forum from where I check out many code snippets to get a grip. As soon as I have this done I can focus more on playing around with strategies.

Some of your points I will have to read up on again and think about them deeper, especially the diversification advice. I know diversification from the classical ETF investing area in terms of asset classes, markets, sectors, etc., but I need to think about how to approach it with algo trading strategies.

Good stuff man, thanks again for your helpful support!

Thom

You seem to have more realistic expectations than I did. Just adding some random thoughts after 4 years of doing this, with some redundancy to the above:

One line version: Beating buy & hold is extremely hard after transaction costs.

Go in with the aim and not the expectation of making money. Only do it if you enjoy it, because you might as well try to become a singer to get rich. In both cases, it boils down to immense effort and quite a bit of intelligence thrown into the mix.

Actually studying academic materials (such as QC and QT provide) might make your trip shorter. (I wouldn't know.)

A bonus IMO is, writing algos can greatly increase your understanding of regular trading. Especially, because it will make you realize that almost 99.9% of TA doesn't give you any statistical edge (at best, it can be used for filtering market conditions). To be blunt, most "traders" you find on forums are lying. The good news is, with a computer you're far more likely to find patterns that have an edge than those attempts approaches they read about in books from authors who made their money in the 70s.

The big elephant in the room is statistical significance since making something that looks great in a test is entirely different from having it working out of sample under real conditions. I found algo trading extremely different from writing computer games which is my own background, because you can't apply an intuitive approach of "if it looks like it works, it works". Counter-intutiively, the more times you checked whether it works, the more likely it is it doesn't...

I must say QC has radically improved since I started on here and the goal to define the necessarily (especially, separating the alpha component which is most the secret sauce) is a great one that lessens the effort necessary to make something that works in practice.

I am not a friend of QC forums' edit lock: I meant to write "necessary modules" above in addition to various misspellings.

Great Discussion !! 🚀 Thank you!