Hi Jared and QC-Team, 

I want to share my opinion regarding the Alpha Market with you. First, I would like to emphasize that I like the idea of the Alpha Market and I hope it will attract more investors and quants in the future. But there are also a few things that bother me. To be constructive with my criticism I will propose possible solutions as well.

 

1. Submission criteria encourage developers to overfitting

In this I refer to the two performance criteria “PSR>80%” and “quick drawdown recovery”. I understand why they are included, but I fear it could have the opposite effect. The concern arose from my own experience. 

I created several good strategies with Sharpe Ratio > 1, low volatility, low beta, small drawdowns etc. and most importantly, I didn’t try to optimize/fit to historical data and needed only a few weak assumptions, so I was confident they would produce similar results in the future. However, it often just missed at least one of the two requirements, so I started adjusting the algorithm. After a while, it met all the criteria for Alpha Market, but I was no longer satisfied with the result. I realized that I was overfitting the strategy.

I then decided against submitting the alpha. However, I doubt that everyone is that self-critical and that makes me doubt the quality of the alphas in the market. 

Long story short, the two performance criteria “PSR > 80%” and “quick drawdown recovery” force many developers to overfitting, which in turn misses the point of the whole thing. 

Suggestions for 1.: I would relax the performance-related criteria a bit and, in return, introduce other criteria that, as an overall package, ensure that only interesting alphas enter the market. More on this later. 

 

2. Brokerage model, margin requirements and regulatory affairs

Currently, the Alpha Streams Brokerage Model is mandatory. This only allows a leverage of 1 for equities, i.e. trading on margin is not possible. 

This made sense in the earlier version of the Alpha Market, where it was exclusively designed for professional investors. These then consumed the insights. But in the current version of the alpha market, it makes less sense in my view, because here the capital allocated for the alpha is managed entirely by the algorithm, including position size/ portfolio weights. 

I think it would be better to use a brokerage model that is close to the broker used for live trading. 

For equities this would typically be 50% as margin requirement (or leverage = 2). 

Incidentally, currency pairs and futures can also be traded with margin, so that equities are treated differently here. The reason for this is not clear to me. 

About regulatory affairs, just two examples: 

Most US-ETFs are not tradable for retail traders from Europe. However, many Alphas trade these ETFs (SPY, TLT etc.), so there are errors in live trading.

Another examples is the pattern day trading rule (PDT) for small accounts (NLV < $ 25k). Again, errors could occur in some circumstances during live trading. On the other hand, this rule does not exist for European markets. 

I am aware that QuantConnect cannot model all regulatory requirements from all countries, but it would at least be good to be able to see in advance what requirements an account must meet in order to license an Alpha for live trading without restrictions and without errors. 

Are there any concrete plans to do something in this regard? 

 

3. Lack of standardization in the Alpha Market

The criteria for approval of an Alpha have changed over time, but old Alphas have remained unchanged. This seems a bit unfair to the developers who joined later. 

I’d appreciate it if all Alphas in the market have to meet the same minimum requirements. 

Developers of older Alphas could be given some time to adjust their strategies if necessary so they can stay in the market. 

But if they don’t meet the criteria after the deadline has passed, they should be archived. So you would still be able to see them for transparency reasons, but they would no longer be licensable/investable. 

 

4. Minimum activity level of Alphas

Currently, at least 100 insights are required. However, insights do not have to result in orders. And also the specification of a minimum number of orders is not effective from my point of view, because then one could simply execute pseudo trades (buy a single share and then immediately sell it), so that the turnover remains low, but the number of orders increases quickly. 

I think a better approach is to require a minimum portfolio turnover or a minimum average asset sales volume. That would be harder to manipulate. 

 

Further suggestions:

  • Minute resolution data or higher (provides more realistic order fills)
  • No order fills at stale prices and Settings.StalePriceTimespan should be reduced to 1 minute (I think default is 10 minutes)
  • Largest drawdown in backtest should be less than 30% (I am surprised that something similar is not already included)
  • Recovery period for drawdowns > 10% must be less than 12 months (currently < 6 months is required, but I think this is too restrictive)
  • PSR > 50% for each rolling 5-year period and SR > 1 for total period (I already commented on this on Slack. Using a rolling metric would also have the advantage of making the result less dependent on the choice of start date. In particular, the criterion “Backtest start date cannot be earlier than 7 years ago” should be omitted. A longer backtest is basically better because it covers more different market regimes.)
  • Turnover > 20% (or an analogous criterion using average asset sales volume ratio)
  • Skin in the game should be rewarded. Quants who are invested with a significant amount (e.g. > $ 10k) in their Alpha should be given special recognition in some form. Skin in the game is an easy way to spot good things.
  • Manual adding of securities should be forbidden (no hardcoded tickers) to avoid selection bias. (Currently not feasible because for ETFs you need some background info like a list of constituents, asset class of constituents, weighting of holdings in ETF etc.)
  • After approval the Alpha should maintain a good reconciliation score (e.g. DTW < 20%)

Let me know what you think.

Someone once mentioned that the overly restrictive requirements were one of the reasons why Quantopian failed, and I think the person is right about that. 

To be more precise, we should very well set high standards, but not in terms of backtest performance, but in terms of resilience and closeness to reality. 

Thank you for your time!