Alpha Stream Scoring Criteria

I know there's countless things that could be added into the fitness function and it's important to limit it to what's crucial and I think you've done a great job of that. I think that there should be a variable of time backtested added. Knowing a history has a significant amount of time that it has been tested on can help bolster some confidence that it will continue in the future. Just my two cents!

Thank you SLPAdvisory! We thought the same, the way we concluded to add time factor using "Portfolio Turnover Ratio x Years of Backtest" would give a ballpark indicator the period was sufficient. We're pushing this today in the backtesting view. e.g.

What is the fitness ranking cutoff for alpha streams approval? 

How is the high turnover ratio determined?

My fitness score and alpha rank are not updating for a certain algorithm.  Any idea what is causing them to be zero?  It was updating on a different algorithm I was using.

Hi Kieran Flynn , it is a "pass" if the average daily turnover of your strategy is greater than 1. This would mean with a $10,000 portfolio you'd need to do at least $10,000 in volume per day. 

I am concerned this might be too hard for more people to hit, and perhaps a more realistic target should be at least 30% portfolio turnover. Currently, the score is linearly scaled down to 0 based on this percentage. So if you only do 5% turnover per day it would scale score to 5.

Jon D. Mitchell just asked a question here which I think might be because of the strategy volume.

The key goal is 100% confidence the returns of the strategy are due to your alpha.

Jared,

Thanks for sharing these ideas with the community and requesting feedback.

Sortino and RoMaD are very good risk metrics.  The volume factor is concerning because it could easily be manipulated and doesn't necessarily provide a fair evaluation of the trading strategy.  

100% daily turnover, or any arbitrary daily turnover rate, favors intraday strategies that may not have desirable risk adjusted returns.  See the following 2 illustrative examples. In my opinion, most people would prefer the risk profile of the biweekly strategy even though it's score is lower than the intraday one.

1. Intraday:  sortino of .35; RoMaD of .25, volume of 1; Score = [1 * (.55+.39) + 10]/20 = .55

2. Biweekly:  assume portfolio only turns over once every 2 weeks (or once every 10 trading days).  Sortino = 2; RoMaD = 1.5; volume = .1; Score = [.1 * (2.67 + 2.14) + 10] / 20 = .52

Along the same lines as SLPAdvisory's suggestion, instead of volume, maybe a better approach would be to look at the risk metrics (Sortino and RoMaD) over a longer period of time, say 5 years, and penalize any strategy that does not provide at least that many years of backtest and/or live data.  For example, fitness = MIN(1, Trading Days/5 years) * Sortino over MIN(Trading Days, 5 years)

It also seems like the range of results is narrow. Most people would be very happy with a strategy that has a Sortino of 1.5 and RoMaD of 1 over a 5 year period; but the score for that would be .68. As a comparison, from 1/1/2014 to 12/31/2018, SPY had a Sortino of .83 and a RoMaD of .45 (assuming the risk free rate = 0), this would result in a score of .6; not too far off from the strategy with much better risk metrics.

My suggestion would be to provide each risk metric as individual filters (one for Sortino and one for RoMaD) on the Alpha stream page. Another metric that may be helpful is total return, or excess return over a risk free rate, to filter out strategies that have good risk measures but low total returns.

There will never be a perfect solution, but our clients are likely sophisticated and value having the ability to filter strategies based on their preferred metrics.  Have you considered asking some of the hedge or fund managers their view?

Thanks again.

I agree with the above comment a lot, I'm not a huge fan of the volume factor. I see it's merits which is why I really like the idea of having the score in all categories shown on the alpha page, but at least for our practice we utilize pretty infrequent trading strategies for the tax benefits as well as limiting trading costs. I have a feeling at least on the RIA advisor side they would be significantly more interested in less active strategies.

Thanks for the comments guys! And an interesting point about manipulating it by simply re-buying SPY each day. 

Just to be clear: funds have the insights, statistics and equity curve and its all delivered by API. The charts on the market page are for the community. The Alpha Streams goal is to allow automated scanning of the marketplace historical results and license in bulk. All of the algorithm's live signals are streamed by API as well. 

The fitness function is for QuantConnect. We are getting a lot of algorithms and what makes it into the market vs what gets bounced is too gray at the moment. I'd like to create a programmatic cut-off so the community can know what the market place is seeking. Imagine we can only host 1,000 alpha stream strategies -- which 1,000 should make the cut? How do you distinguish without imposing bias?

Ultimately the goal of the volume component was to programmatically determine the strategy is trading in a statistically significant manner so we can attribute the performance to the owner. For HFT strategies "statistical significance" can be seen in a 3 month backtest so we don't want to impose a period like 5 years into the equation.

Do you have other ideas for filtering out the "Buy APPL in 1998" strategies? 

Re: SPY vs Intraday Tristan -- one community member suggested in private feedback to multiply the values of RoMaD and Sortino, and scale to the new distribution. This way the poor RoMaD of SPY would eliminate it as a high drawdown strategy. 

Also important to mention -- the fitness score running in production is slightly different to the one in the research notebook above - the key difference being Martin shifted the scaled values by 5 - this allowed algorithms which have an absolute value of 0 (e.g. doing no trading, or losing money) to start from 0 instead of 50%! 

The test I usually perform to account for overfitting or what my cover the "Buy AAPL or AMZN in 1998" is to just change the ticker. Especially if it's a intraday algorithm that is analyzing patterns or trends, I'm of the camp that it should be able to be applied to similar success across tickers. You could try substituting the ticker in the provided algorithm with a basket of indexes? Really glad to see your involvement with the community on this!

Jared Broad: Do you have other ideas for filtering out the "Buy APPL in 1998" strategies? 

I have got an idea. It is aimed more at the composition of the portfolio than the turnover. 

In my eyes a portfolio with a lot of different open positions is much more resilient than a portfolio with AAPL stocks only, especially when the open positions are little or even negative correlated to each other. 

where N is the number of open positions and c_ij are the correlations coefficients between each open position. So C is nothing else than the sum of all elements of the correlation matrix except the diagonal. To divide the sum by N^2 makes sure C is between -1 and 1 so it should be easy to shift the fitness function. 

The smaller C, the better the portfolio. 

By subtracting C from your current fitness function you punish "Buy AAPL in 1998" strategies with a worse score. 

On the other hand, a portfolio with a lot of negative correlated positions would get a better score. That's a good thing in my eyes because a portfolio with more negative correlated positions is less vulnerable in market turmoils.

Re: I've to admit that correlations can change with time but usually they persist for a long time. However, we should use something like the rolling 90 day correlation instead of using the whole price history. 

"Buy AAPL in 1998" yields 0 Fitness Score currently because there's zero turnover. I think these kinds of strategies will naturally ruin their own scores - it's hard to "cheat" meaningful performance with the kind of turnover needed to get a high score.

It's anyway highly likely that it will be necessary to manually reviewing every strategy meeting the initial score cut-off. "Cheaters" can be caught with relative ease this way I suspect.

Having said that, I don't think the current implementation of GetScaledPortfolioTurnover() is sound - you're rewarding strategies that turn over themselves entirely every day, or close to that, bringing about enormous trading costs and throwing any kind of scalability out the window. I'm assuming you have had advice from funds that this is what they look for, but frankly I'm really surprised this is implemented in this way.

Large quant institutions who will be the primary licensors have a bias towards active strategies as they can be quickly switched off when performance out of sample changes. However, the turnover was intended to be a minimum turnover for acceptance, I was concerned 1.0 might be too hard for most people to hit but a lower minimum of 0.3 might be more suitable. The linear function 0-1 @ MinTurnover is not designed to incentivize/reward any strategy but set that minimum volume, if the volume is a good signal for significance.

Thank you for the suggestion Arthur. We can't assume it is a portfolio based strategy - if you have an alpha which predicts orange juice futures alone its still a great factor for a multifactor portfolio. However, building off your idea perhaps we could use the correlation of the open positions to the equity curve. This would be a direct way of measuring the performance is due to the assets. 

Tristan F  -- SPY with Portfolio turnover of 2.0 -- score of 55.

Jared,

You can create a volume factor of 100% by being out of the market for 1 minute instead of over night.  See attached for score of 60.

Is the Sortino ratio available anywhere on backtest results?  I can't find it.

Thanks.

Right! And that score is actually 200% volume factor as each side of the trade contributes. This is why we set it to 1.0 to start with - its an easily achievable goal if you're actually trading. 

Thanks for clarifying Jared.  If you believe your clients will be focused on HFT trading type algorithms than maybe the volume factor works.  I still believe it will leave out some very good algorithms and that it's too easy to manipulate to be very useful.

Another point is that Sortino and RoMaD are measuring very similar things.  Sortino is return over negative deviation, and RoMaD is return over maximum drawdown, One measures the expected and the other the extreme downside risk.  What's the reason for including both?  Are there well known use cases where Sortino looks great but the RoMaD is terrible, or vice versa?

I think Sortino's negative deviation results in it being an average negative variance of the population (sum/count) which lets it hide martingale type strategies with 99% perfect trades and one 50% drawdown. The reason for including both was so that RoMaD would eliminate the absolute worst value drawdown "martingale-option-put selling" hidden risk algorithms.

Hi,

I'd add a criteria to this Fitness Function which would take into account the long running well performing backtests (see example below). Does it make sense? What do you think?