Analyzing the backtests results issue

Loading web-font TeX/Math/Italic

QuantConnect Community Discussions

QUANTCONNECT COMMUNITY

Join Our Discord Channel

Join QuantConnect's Discord server for real-time support, where a vibrant community of traders and developers awaits to help you with any of your QuantConnect needs.

Draft Discussions

Bookmarked Discussions

Analyzing the backtests results issue

Last comment by - November 2020

Analyzing the backtests results issue

Discussion on analyzing backtest results and need for more log data. Suggests extracting valuable info from trades data. Seeking input.

Share New Research

Start New Discussion Sign up

SEARCH DISCUSSIONS

361,000 Quants.

Become a Quant

VOTE FOR UPCOMING FEATURES

Share your input and vote on our future direction.

LEAN Roadmap

JOIN OUR Community MAILING LIST

Create an account on QuantConnect for the latest community delivered to your inbox.

Analyzing the backtests results issue

JayJayD | November 2015

Hey Folks:

These days some posts were about the problem of analyzing the backtests results, and the need of more log.

After thinking about it I realized that the engine already gives us all the information we need to fully analyze the backtest in the downloadable trades log. But that data need to be processed in order to obtain valuable information.

What can be considered valuable information? To begin, just the basics: the daily algorithm equity curve, the daily volume traded by stock, the daily result by stock. Maybe you can help me naming some others?

I see two options to extract the information from the trades data:

- Make a program to process the trades data.

- Make the engine generate the info and make it available just like the trades data: I think this is a better option because the engine already estimates those values through the rolling statistics. Or, at least, the info is easily available from Transaction and Trade QCAlgorithm properties.

So, what do you think?

I’m available to help in the development of this issue.

Author

JayJayD

November 2015

Upvote

3 people upvoted this

Jared Broad INVESTOR

QuantConnect | November 2015

Awesome @Juan, I definitely agree all the data is there and its just a matter of processing it into nice charts and picking out which charts we'd like :) Even better than the downloadable trade log is the full BacktestResultPacket -- this has the equity charts, daily profit loss and all the statistics you'll need. We want to make it so people can use this JSON object and load up all the information for a backtest. I'd recommend we write extensions / enhancements to the Statistics class. This way the results will also be available on the website virtually immediately. Alternatively if your work requires special charting libraries you could add it to the toolbox project. I like the colored daily/weekly distribution matrix idea, larger rolling sharpe, rolling drawdown charts (we don't have this calculated), top drawdown periods, distribution of returns. We need some more infrastructure to do "out of sample" charting.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Upvote

JayJayD INVESTOR

November 2015

Hey @Jared, I really want to help but the topic is way above my head. I don’t even know what JSON is and certainly I can’t do any of the task listed here. I tried to figure out how the things works in debugging mode, and I quickly get lost in so many classes and calls. Thinking out loud: I think I can make the calls to estimate the daily volume traded per stock but I’m not sure how that calls can fit in the engine. Anyway, the equity data should be available to download as csv just like the trades logs; even if isn’t the best option.

Upvote

Jared Broad INVESTOR

QuantConnect | November 2015

Hey guys, you can now download the JSON object which generated your backtest result :) Hopefully this empowers you to make beautiful dashboards with even deeper analysis. You can do this by clicking on the "Download Result" button in the Share tab. This object can be deserialized into a C# type 'BacktestResultPacket' using Newtonsoft.Json.

var resultJsonString = File.ReadAllText("my-backtest-result.json");
var results = JsonConvert.DeserializeObject(resultJsonString);

Upvote

2 people upvoted this

JayJayD INVESTOR

November 2015

WOW! Just great. Thank you very much Jared and all the QC staff! I was working in the calls I mentioned above, so far this is what I have:

var dailyStatisticsByStock = TradeBuilder.ClosedTrades
                           .GroupBy(trades => trades.Symbol,
                                    (symbol, trades) => new
                                    {
                                        ticket = symbol,
                                        dailyValues = trades
                                        .GroupBy(trade => trade.ExitTime.Date,
                                                    (day, trade) => new
                                                    {
                                                        index = day.Date,
                                                        profit = trade.Sum(t => t.ProfitLoss),
                                                        volShares = trade.Sum(t => t.Quantity),
                                                        volDollars = trade.Sum(t => t.Quantity * (t.ExitPrice + t.EntryPrice))
                                                        // Any other daily statistic by stock.
                                                    }
                                                    )
                                    });

Is pretty ugly, and I'm not sure if the volume is correctly estimated, but is a first working draft. Some of you will note the Pandas Panel way of thinking, i.e. there is a DataFrame for each symbol with the daily statistics. As you can see only needs the TradeBuilder, so as far I understand, it can be called from anywhere in the engine. I'll research how to correctly make the volume estimations the and how to serialize this object to JSON. Cheers, JJ

Upvote

Jared Broad INVESTOR

QuantConnect | November 2015

Percentage of time in cash could be interesting, perhaps this can be calculated from the trades. We have a daily performance gain/loss chart under the name "Daily Performance". This could be used for summing monthly PnL.

Upvote

JayJayD INVESTOR

November 2015

Hey @Jared, In order to estimate the percentage of time in cash, I'm thinking in use the Time Period library (available in NuGet); more specifically, the TimeGapCalculator . The idea is defining a time period for each trade in the TradeBuilder.ClosedTrades using the EntryTime and ExitTime and estimates the gap. Once the gap is estimated, is jut matter of make a division to obtain the percentage of time in cash. What do you think? Edit: I must incorporate all the open trades too!

Upvote

Bruce Vanstone INVESTOR

November 2020

Can you please give a python example of how to do this... and what the name of the backtest object is after I have run a backtest.

I am quite happy to contibute time to developing advanced analytics, but it is unclear how to obtain the data to start with,

cheers,

Bruce

Upvote

1 person upvoted this

Derek Melchin INVESTOR

QuantConnect | November 2020

Hi Bruce,

To get the backtest results into the research environment, we can use

backtest = api.ReadBacktest(projectId, backtestId)

We are working on tools to help process the $backtest$ result object. For now, we can access the Sharpe ratio for example with

backtest.Result.TotalPerformance.PortfolioStatistics.SharpeRatio

See the attached research notebook for an example.

The backtestId can be seen under the "Share" tab of the backtest results page

Best,
Derek Melchin

Upvote

1 person upvoted this

Tzuo Hann Law INVESTOR

ECON3382 | November 2020

Hi I just tried to replicate this, but it did not work. Can someone confirm that this is functional please?

Upvote

Arthur Asenheimer INVESTOR

Monetark | November 2020

It didn't work for me, too.

The commands api.Connected and backtest = api.ReadBacktest(projectId, backtestId) were successful, but backtest.Result yields a NoneType object.

Upvote

Erol Aspromatis INVESTOR

November 2020

I'm getting the same error, the API is not finding my backtest and retuirning it, rather it is creating a empty object.

AttributeError Traceback (most recent call last) <ipython-input-15-b8bc2d3efee3> in <module> ----> 1 backtest.Result.TotalPerformance.PortfolioStatistics.SharpeRatio AttributeError: 'NoneType' object has no attribute 'TotalPerformance'

Upvote

Jared Broad INVESTOR

QuantConnect | November 2020

Sorry about we made some fast-breaking changes before the holiday weekend and will push the fixes for them today. We move quickly and there will always be bugs in complex software. Microsoft is still patching Windows since 1989 =). But please respect the forum etiquette and not post bug reports to the forum -- please submit all bug reports to support@quantconnect.com so we can keep the forum focused on algorithm development.

Upvote

1 person upvoted this

Jonathan Evans INVESTOR

December 2015

So, let me make sure I understand this: (1) The universe is a set of symbols, defined by a user-supplied filter function. (2) Every (UniverseSettings.Resolution) time span, the universe will be updated by calling the filter function. (3) If and only if the set of symbols changes, OnSecuritiesChanged is called and given a SecuritiesChanges object. Is that an accurate summation? Any potential gotchas or limitations we should be aware of?

Upvote

Michael Handschuh INVESTOR

December 2015

1. Absolutely spot on! A little more specific though, in general, a universe takes in some data (in this case CoarseFundamental) and returns a filtered set of symbols. In another forum post I demonstrated how you can use your own custom data sources for universe selection data! There also exist overloads that only accept a DateTime argument and return a set of symbols, useful for hitting an external API that doesn't lend itself to our custom data system. EDIT: Here's a link to the other forum post where I shared a custom data universe example. 2. Correct, more specifically, it's every resolution step within the market hours of the universe's market. For instance, in the example project attached it is a US-Equity market universe, so it will fire a midnight the day after trading, so after Monday's trading, Tuesday at midnight all the way until after Friday's trading which is Saturday at midnight. Likewise, for minute data (no extended hours) it will start firing at 9:31AM and continue until 4:00PM NewYork time. 3. Correct, OnSecuritiesChanged will only fire if the securities have actually changed. An interesting point is that you can define multiple universes. If this is done, it will contain all the changes (set union) between the two universes. Another important note pertaining to multiple universes is that one universe can't remove another universe's securities. One thing that will soon be implemented is the ability to define the minimum time a stock will exist in the universe. This will be useful if you require indicators or state that require multiple days to become usable. I expect to have this ready before the end of next week. We've held off on pushing this feature until it was pretty solid, so if you find anything you wouldn't expect please let us know. You may enjoy taking a peek at the various overload here that offer varying degrees of flexibility. For ultimate flexibility you could derive from the Universe type directly, but that's pretty advanced :)

Upvote

Max_leverage_yolo INVESTOR

December 2015

so we can simulate the s&p500 by setting top 500 by market cap?

Upvote

Michael Handschuh INVESTOR

December 2015

The coarse data set does not currently include a figure for market cap, but you can simulate the sp500 by taking the top 500 stocks by daily dollar volume.

Upvote

Nat Miletic INVESTOR

January 2016

Great work Michael. Will the list of universe selection data points be expanded at some point? Trying to back test the Dogs of the Dow strategy and require yield. Thanks!

Upvote

Jared Broad INVESTOR

QuantConnect | January 2016

@Nat -- Yes! We started with this simple data-set but will expand it with other fundamental values in the next few months.

Upvote

Travis Teichelmann INVESTOR

February 2016

When I add the top method to Initialize I receive a message that OneBillion doesn't exist in the current context. I've tried changing it to an integer and looking for members that in DollarVolume property but couldn't find anything. Any thoughts?

Upvote

Michael Handschuh INVESTOR

February 2016

You can add const decimal OneBillion = 1000m*1000m*1000m;. It's just a constant to find stocks with volumes over a billion, Likewise, you could change the volume filter to anything you like!

Upvote

1 person upvoted this

Travis Teichelmann INVESTOR

February 2016

Perfect! Thank you for updating the code. I modified what you provided to filter the securities based on price. Now all I need to do is implement a gap up and gap percentage method then I can move on to other parts of my algo. Here's what I did.

Upvote

2 people upvoted this

Stephen Oehler INVESTOR

February 2016

So this might be a dumb question, but can you apply indicators to the coarse universe selection function? Wondering if I could say, for example, "grab all stocks whose 1-year momentum percent is greater than 10%". :-) Excellent work guys!

Upvote

Stephen Oehler INVESTOR

February 2016

Ah found it in an older thread, sorry about that! For those who come here wondering the same thing, here is how you conduct technical analysis in the Coarse Universe filter: https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/EmaCrossUniverseSelectionAlgorithm.cs

Upvote

1 person upvoted this

Travis Teichelmann INVESTOR

February 2016

Stephen - You can use indicators on the universe selection, that's actually what I'm working on right now. I've been using the link you provided in another post. https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/EmaCrossUniverseSelectionAlgorithm. Community - I've been looking at the LogReturns indicator and found a way to compare the difference in the current price with yesterday's close. Translating those findings into the coarse universe selection is a bit more tricky. Similar to what Stephen was saying, I want to scan the entire universe for stocks that have gapped up. Then scan those to find ones that have the highest gain. This is one of the biggest parts of my strategy and I've been working on it for a while. So if you could point me in the right direction or give me an example that would be fantastic. Best, -Travis

Upvote

2 people upvoted this

Michael Handschuh INVESTOR

February 2016

Hey Travis, you don't need to copy all of your indicators into the project. They should be available without adding the file. Sadly the gap open condition requires open and close prices, but the coarse universe data set only has the daily closing price, so I'm not sure you can detect gap up/down on open. If you're just looking for stocks that had the largest one day gains, take a look at this example algorithm. I used the WSJ top gainers page to find stocks that gained the most in one day... and then shorted them!

Upvote

1 person upvoted this

Stephen Oehler INVESTOR

February 2016

Hi Michael, Quick question for ya: If you have indicators that are being used within the Coarse Universe Linq statement, are your indicators updated daily or are they updated at the pace of the Universe Resolution? Example of the indicators being declared and updated within the Linq statement: https://github.com/QuantConnect/Lean/blob/master/Algorithm.CSharp/EmaCrossUniverseSelectionAlgorithm.cs I'm getting too-good-to-be-true results in my algorithm. It uses a certain indicator that requires 20 days of daily data. My Universe resolution is set to minute though, and I'm wondering if I'm getting a self-fulfilling prophecy problem here (maybe benefiting from fill-forwarding?).

Upvote

Travis Teichelmann INVESTOR

March 2016

Micheal - Can you backtest live data or is it exclusive to live mode? I've tried to run this algorithm live and made sure the links to WSJ were functional though no trades are being made. I was also inspecting the HTML on the site to find more data to parse. It doesn't seem to be pulling in any data even after I left it running overnight. Trying to run a price filter where d.Price

Upvote

Stephen Oehler INVESTOR

March 2016

Hi Travis, I thought I saw someone use custom data with their backtest. Is that the same as "live data" in this case? https://www.quantconnect.com/forum/discussion/418/bubble-algorithm-using-cape-ratio-macd-and-rsi#Item_12 Hopefully I'm not misunderstanding you!

Upvote

Michael Handschuh INVESTOR

March 2016

When using indicators in any universe selection function, they will be updated at the same frequency as the universe gets updated. In the case of coarse universes, they will update on a daily time frame. Another thing to note is that we do not apply fill forward to universe selection. If you need more help please post an algorithm demonstrating your issue/concern.

Upvote

1 person upvoted this

Michael Handschuh INVESTOR

March 2016

Travis - I'll run the algorithm over this evening with some debugging to see if things are working as expected. Did you make any changes to the example algorithm? I'm not sure what you mean by 'backtest live data.' The NyseTopGainers custom data type is written to support both live and backtesting cases. It supports the backtest case by hitting dropbox (I made a small script to scrape the pages and built a csv from it)

Upvote

1 person upvoted this

Stephen Oehler INVESTOR

March 2016

Great, thanks for the validation, Michael!

Upvote

Michael Handschuh INVESTOR

March 2016

Clarification on my above statement: We don't apply fill forward to the universe selection data itself, but subscriptions added via universe selection can be set to have fill forward data using the UniverseSettings.FillForward property. This is set to true by default.

Upvote

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

361,000 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

Actions

Join QuantConnect for Free

Platform

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

361,000 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

SHARE

Actions

Join QuantConnect for Free