Analyzing the backtests results issue

Loading [MathJax]/jax/output/HTML-CSS/config.js

QuantConnect Community Discussions

QUANTCONNECT COMMUNITY

Join Our Discord Channel

Join QuantConnect's Discord server for real-time support, where a vibrant community of traders and developers awaits to help you with any of your QuantConnect needs.

Draft Discussions

Bookmarked Discussions

Analyzing the backtests results issue

Last comment by - November 2020

Analyzing the backtests results issue

Discussion on analyzing backtest results and need for more log data. Suggests extracting valuable info from trades data. Seeking input.

Share New Research

Start New Discussion Sign up

SEARCH DISCUSSIONS

360,600 Quants.

Become a Quant

VOTE FOR UPCOMING FEATURES

Share your input and vote on our future direction.

LEAN Roadmap

JOIN OUR Community MAILING LIST

Create an account on QuantConnect for the latest community delivered to your inbox.

Analyzing the backtests results issue

JayJayD | November 2015

Hey Folks:

These days some posts were about the problem of analyzing the backtests results, and the need of more log.

After thinking about it I realized that the engine already gives us all the information we need to fully analyze the backtest in the downloadable trades log. But that data need to be processed in order to obtain valuable information.

What can be considered valuable information? To begin, just the basics: the daily algorithm equity curve, the daily volume traded by stock, the daily result by stock. Maybe you can help me naming some others?

I see two options to extract the information from the trades data:

- Make a program to process the trades data.

- Make the engine generate the info and make it available just like the trades data: I think this is a better option because the engine already estimates those values through the rolling statistics. Or, at least, the info is easily available from Transaction and Trade QCAlgorithm properties.

So, what do you think?

I’m available to help in the development of this issue.

Author

JayJayD

November 2015

Upvote

3 people upvoted this

Jared Broad INVESTOR

QuantConnect | November 2015

Awesome @Juan, I definitely agree all the data is there and its just a matter of processing it into nice charts and picking out which charts we'd like :) Even better than the downloadable trade log is the full BacktestResultPacket -- this has the equity charts, daily profit loss and all the statistics you'll need. We want to make it so people can use this JSON object and load up all the information for a backtest. I'd recommend we write extensions / enhancements to the Statistics class. This way the results will also be available on the website virtually immediately. Alternatively if your work requires special charting libraries you could add it to the toolbox project. I like the colored daily/weekly distribution matrix idea, larger rolling sharpe, rolling drawdown charts (we don't have this calculated), top drawdown periods, distribution of returns. We need some more infrastructure to do "out of sample" charting.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Upvote

JayJayD INVESTOR

November 2015

Hey @Jared, I really want to help but the topic is way above my head. I don’t even know what JSON is and certainly I can’t do any of the task listed here. I tried to figure out how the things works in debugging mode, and I quickly get lost in so many classes and calls. Thinking out loud: I think I can make the calls to estimate the daily volume traded per stock but I’m not sure how that calls can fit in the engine. Anyway, the equity data should be available to download as csv just like the trades logs; even if isn’t the best option.

Upvote

Jared Broad INVESTOR

QuantConnect | November 2015

Hey guys, you can now download the JSON object which generated your backtest result :) Hopefully this empowers you to make beautiful dashboards with even deeper analysis. You can do this by clicking on the "Download Result" button in the Share tab. This object can be deserialized into a C# type 'BacktestResultPacket' using Newtonsoft.Json.

var resultJsonString = File.ReadAllText("my-backtest-result.json");
var results = JsonConvert.DeserializeObject(resultJsonString);

Upvote

2 people upvoted this

JayJayD INVESTOR

November 2015

WOW! Just great. Thank you very much Jared and all the QC staff! I was working in the calls I mentioned above, so far this is what I have:

var dailyStatisticsByStock = TradeBuilder.ClosedTrades
                           .GroupBy(trades => trades.Symbol,
                                    (symbol, trades) => new
                                    {
                                        ticket = symbol,
                                        dailyValues = trades
                                        .GroupBy(trade => trade.ExitTime.Date,
                                                    (day, trade) => new
                                                    {
                                                        index = day.Date,
                                                        profit = trade.Sum(t => t.ProfitLoss),
                                                        volShares = trade.Sum(t => t.Quantity),
                                                        volDollars = trade.Sum(t => t.Quantity * (t.ExitPrice + t.EntryPrice))
                                                        // Any other daily statistic by stock.
                                                    }
                                                    )
                                    });

Is pretty ugly, and I'm not sure if the volume is correctly estimated, but is a first working draft. Some of you will note the Pandas Panel way of thinking, i.e. there is a DataFrame for each symbol with the daily statistics. As you can see only needs the TradeBuilder, so as far I understand, it can be called from anywhere in the engine. I'll research how to correctly make the volume estimations the and how to serialize this object to JSON. Cheers, JJ

Upvote

Jared Broad INVESTOR

QuantConnect | November 2015

Percentage of time in cash could be interesting, perhaps this can be calculated from the trades. We have a daily performance gain/loss chart under the name "Daily Performance". This could be used for summing monthly PnL.

Upvote

JayJayD INVESTOR

November 2015

Hey @Jared, In order to estimate the percentage of time in cash, I'm thinking in use the Time Period library (available in NuGet); more specifically, the TimeGapCalculator . The idea is defining a time period for each trade in the TradeBuilder.ClosedTrades using the EntryTime and ExitTime and estimates the gap. Once the gap is estimated, is jut matter of make a division to obtain the percentage of time in cash. What do you think? Edit: I must incorporate all the open trades too!

Upvote

Bruce Vanstone INVESTOR

November 2020

Can you please give a python example of how to do this... and what the name of the backtest object is after I have run a backtest.

I am quite happy to contibute time to developing advanced analytics, but it is unclear how to obtain the data to start with,

cheers,

Bruce

Upvote

1 person upvoted this

Derek Melchin INVESTOR

QuantConnect | November 2020

Hi Bruce,

To get the backtest results into the research environment, we can use

backtest = api.ReadBacktest(projectId, backtestId)

We are working on tools to help process the $b a c k t e s t$ $backtest$ result object. For now, we can access the Sharpe ratio for example with

backtest.Result.TotalPerformance.PortfolioStatistics.SharpeRatio

See the attached research notebook for an example.

The backtestId can be seen under the "Share" tab of the backtest results page

Best,
Derek Melchin

Upvote

1 person upvoted this

Tzuo Hann Law INVESTOR

ECON3382 | November 2020

Hi I just tried to replicate this, but it did not work. Can someone confirm that this is functional please?

Upvote

Arthur Asenheimer INVESTOR

Monetark | November 2020

It didn't work for me, too.

The commands api.Connected and backtest = api.ReadBacktest(projectId, backtestId) were successful, but backtest.Result yields a NoneType object.

Upvote

Erol Aspromatis INVESTOR

November 2020

I'm getting the same error, the API is not finding my backtest and retuirning it, rather it is creating a empty object.

AttributeError Traceback (most recent call last) <ipython-input-15-b8bc2d3efee3> in <module> ----> 1 backtest.Result.TotalPerformance.PortfolioStatistics.SharpeRatio AttributeError: 'NoneType' object has no attribute 'TotalPerformance'

Upvote

Jared Broad INVESTOR

QuantConnect | November 2020

Sorry about we made some fast-breaking changes before the holiday weekend and will push the fixes for them today. We move quickly and there will always be bugs in complex software. Microsoft is still patching Windows since 1989 =). But please respect the forum etiquette and not post bug reports to the forum -- please submit all bug reports to support@quantconnect.com so we can keep the forum focused on algorithm development.

Upvote

1 person upvoted this

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

360,600 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

Actions

Join QuantConnect for Free

Platform

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

360,600 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

SHARE

Actions

Join QuantConnect for Free