QuantConnect Community Discussions
QUANTCONNECT COMMUNITY
Hello All!
We're looking for feedback on a proposed new Alpha selection filter. The Probabilistic Sharpe Ratio would give us another way to measure the results of backtests submitted and provide funds looking to license Alphas with more information about algorithm performance beyond our current metrics. If you have suggestions please clone the backtest, examine the notebook, and give us your thoughts!
Thanks,
Jack
Upvote
9 people upvoted this
QuantConnectâ„¢ 2025. All Rights Reserved
Share New Research 
Frank Norman
This would lead to higher performing Alpha Streams. I would add it to the arsenal.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Andrew martin czeizler
Is this being included in the backtest metrics?
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jack Simonson
We haven't included it yet into the backtest metrics, but it's something that we're considering as well.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Tim Bohmann
I would like to see this in the backtest metrics. This would be a quick "sanity test" when backtesting.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Pangyuteng
Glad to see that Quantconnect seems to be agile and willing to update methods for their algo selection. I also find it comforting that Lean is opensource, so I can check the implementation and their test cases. :)
SPY notebook SR: 0.836, PSR: 50.000% Lean SR: 0.900, PSR: 41.185% ---- SHY notebook SR: 1.237, PSR: 68.901% Lean SR: 1.224, PSR: 69.799% ---- QQQ notebook SR: 0.910, PSR: 52.476% Lean SR: 0.990, PSR: 49.063% ---- AMZN notebook SR: 1.218, PSR: 83.669% Lean SR: 1.339, PSR: 78.661% ---- GE notebook SR: -0.458, PSR: 10.897% Lean SR: -0.306, PSR: 0.159% ----For those that want to do some quick and dirty backtest or optimization/investigation in research, please do keep in mind the inevitable computation differences between LEAN and simple PSR computation in notebook. I think this is one of those little but important things most engineers/data scientists have to deal with when working among different environments. Anyways... per Alexandra's response, "rolling PSR" is computed by LEAN (I can't seem to figure out where the rolling part is though in github), while Jack's PSR is a single value derived from one stream of returns. Below shows you the difference for Sharpe Ratio (SR) and PSR for a few tickers if you buy-and-hold in the same timeframe between notebook (using annualized SR) and Lean. I would expect PSR for SPY to be a bit closer to 50% from LEAN. Avid notebook users... beware!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Kamer Ali YUKSEL
I would be more than glad if someone can also share how to calculate rolling PSR.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Kamer Ali YUKSEL
I also have a question, why you haven't utilized Deflated Sharpe Ratio? That seems to be an extension over PSR.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
DSR was not deterministic, and required analysis of many backtests. PSR was chosen because it could be self contained with a single analysis.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Kamer Ali YUKSEL
Hello Jared, thanks for letting me know.!I have a small objection about PSR. Since that it uses a long-only SPY benchmark, it does not make perfect sense for long-short portfolios or a long-only ETF portfolio that includes some inverse ETF(s), such as the current universe. I sometimes obtain better results in many of the metrics but the PSR is lower. To sum up, it is not a very good metric in terms of taking the market-neutrality into account (except for a long-only portfolio that does not include any Inverse ETFs - which is not valid for the current competition universe).
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
We chose not to use SPY as the benchmark but a fixed Sharpe-ratio of 1.0 to make the measurement cross-asset / cross-strategy type; so the PSR readings in LEAN's case are the probability the real algorithm returns are greater than 1.0 Sharpe ratio.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Kamer Ali YUKSEL
Just to let you know, I encountered another implementation of PSR in some other source codes that I have checked. The first one below is that other implementation and the second one below is the implementation in this thread. I did not yet compare the output of those but they seemed to be different to me. I would welcome an advise on this.
std = np.sqrt(1 - (sps.skew(ret)*sharpe) + sharpe**2 * (sps.kurtosis(ret)-1)/4)
return sps.norm.cdf(sharpe * np.sqrt(len(ret)-1) / std)
std = 1 - (sps.skew(ret) * sharpe) + ((sps.kurtosis(ret)-1)/4) * sharpe
return sps.norm.cdf(sharpe / np.sqrt( std / (len(ret) - 1)))
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Rahul Chowdhury
Hey Kamer,
The other implementation of PSR you provided is actually the correct implementation as defined by the paper The Sharpe Ratio Efficient Fronter by Bailey D. & Prado M. (2012). That equation is the one that is defined in LEAN and is the correct one. However, we made a small mistake in copying it over in the journal post, which we've since fixed. There was a missing factor of Sharpe in the kurtosis term. With the added Sharpe factor, the two equations are the same, just differently represented. Thanks for pointing it out!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Kamer Ali YUKSEL
Hello Rahul, Yeah, I also verified that later but did not want to write it here to make a conclusion :) Thnx!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Apollos Hill
So i am finally rereading this thread. Sorrry. I know its kind of old now and has been added ot the Alpha criteria. I'm looking for tips on how to increase my PSR past 50 .
" the Probabilistic Sharpe Ratio is an atemporal measure of strategy performance expressed in terms of probability of skill beyond a given benchmark..." If we are using 1.0 Sharpe as a benchmark and not SPY...if my strategy is doing 250% return over a 5 year backtest, how am i only at 25-50% PSR and 0.8 sharpe? IS it suggesting that I somehow am not maximizing the profit potential of my strategy? I don't want to post a backtest here but i am in forex, trading 4 out of the 8 major pairs. I can't trade all the pairs or else i'm over exposed. I'm trading on weekly bars and holding for months using million dollar positions. I'm also in positions as much as i can be without over trading. 250% over 5 years is about 50% annual returns. Is that not worthy of a 1.0 sharpe? I doubt the assets that Ted posted above (with LEan SR thanks) could achieve these returns.
Appreciate the help on increasing my PSR so i can lease this alpha stream.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jared Broad
Hi Apollos, it means the volatility of the strategy is too high. Perhaps you can invest smaller, or risk-adjusted positions somehow? Could you invest for shorter periods to help reduce the exposure to swings of the market?
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Apollos Hill
Hi Jared
I tried only investing 1% per trade. I'm in the process of seeing if i can go to daily bars instead of weekly. Right now i have a nice 45degree slope up. It doesn't seem volatile to me. My Annual Variance and my Beta are 0.00. Not sure if that helps you or not. my sharpe is 0.729 so i guess PSR at 25% is saying that i have a 25% chance that my sharpe will be over 1% . So i need to improve my money making, without increasing drawdown, correct? 10% DD on a 48% total return over 5 years. (nowthat i changed to 1% positions)
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Rahul Chowdhury
Hey Apollos,
There is currently an issue with beta calculation for daily resolution data which produces 0 beta for all strategies. The true beta of your strategy may actually be larger. If your strategy is trading the same symbols for the same duration, changing position sizes will not dramatically change your PSR because your returns and drawdowns will scale proportionally. Instead try shortening your insight period or changing how you emit insights.
> my sharpe is 0.729 so i guess PSR at 25% is saying that i have a 25% chance that my sharpe will be over 1%
It means there's a 25% change your live-Sharpe is greater than 1.0; (in part because the backtesting Sharpe is 0.7)
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Ritaelyn
Hi Jared,
What's your thoughts of allowing for a sortino ratio as another means for submission? One thing I'd like to submit to alpha is about 3.60 sortino but it's under 1.0 sharpe, as sharpe unfortunately penalizes both upside and downside volatility equally.
Thanks for considering it!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Hector Barrio
Hello Ritaelyn, I agree with you. Any equity trading during the COVID instability will be highly penalized by SR/PSR metrics, even if properly hedged and/or profitable. The normality assumption of the sharpe ratio can, in extreme cases, make very large positive returns in possitively skewed strategies undesirable. It is also a possibilty to have strategies that could "fish" for higher PSR by extending the test period to low volatility-low return periods (cool periods) even at equity loss.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
.ekz. INVESTOR
+1 to Ritaelyn point. I'd love to see Sortino Ratio. From what I've been reading, sharpe ratio (and subsequently PSR as well) is not the best perf indicator for trend following strategies.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Shile Wen
Hi .ekz.,
Users interested in metrics not supported by web QC can save the portfolio values and do custom calculations in OnEndOfAlgorithm and print the results or retrieve the results of the algorithm as shown in this post.
I've shown how to the former method in the attached backtest.
Best,
Shile Wen
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jake Sehnert
Is the PSR calculated using CSCV?
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Abhijeet Mulye
Longer duration backtests have higher PSRs. Is there a standardized time period for making sure an alpha has >= 80% PSR before submitting?
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Derek Melchin
Hi Jake,
No cross validation is used in the calculation. The formula is available in the notebook at the top of this thread. Implemented in Python, it is:
def get_psr(self, benchmark = 0): skew = sps.skew(self.returns) kurtosis = sps.kurtosis(self.returns) n = len(self.returns) estimate_std = np.sqrt( (1 - (skew*self.sharpe_ratio) + ((kurtosis-1)/4)*self.sharpe_ratio) / (n - 1) ) return round(sps.norm.cdf((self.sharpe_ratio - benchmark)/estimate_std),3)Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Derek Melchin
Hi Abhijeet,
When submitting an alpha for Alpha Streams, the strategy must be backtest over the most recent 5 years. Refer to our documentation here.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!