Strategy Optimization

Hi there,

Based in the awesome work made by James Smith and this tutorial, I implemented a project called LeanSTP, which is able to run multiple Lean instances in parallel.

The only significant difference respect to the work done by James is that the parallelization is outside the genetic algorithm environment, the output is saved as JSON and the Log file is saved for each run. The default folder is Public Documents

The core to configure the parameters lays in this method. In this the example, the algorithm runs some combinations of EMA periods and runs multiple instances of the ParameterizedAlgorithm.

But IMHO, optimizing backtests is a bad idea. Is relatively easy to find an optimal strategy, even for a pure random walk, and The higher the number of configurations tried, the greater is the probability that the backtest is overfit. I guess you can calibrate some parameters without hurting so much the out of sample performance.

Anyway, I hope someone find this helpful.

The papers you've highlighted are very interesting, and cross over into some of the research areas I've been tackling. I think the GeneticSharp library that my optimizer references uses a similar approach to the smart thread pool in order to achieve parallel backtests. It's useful to have a trimmed down execution model for just this task.

In terms of the concept of backtest overfitting in general, this is certainly a problem. I would not recommend trusting an optimal result from just in sample data: rather its advisable to take multiple in samples and/or validate against an out of sample set. It's for this purpose that I've recently completed work on an optimization Batcher, that will automate the process of walk forward testing across multiple periods.

On a more general note, I think whilst there is not a (proven) efficient alternative, backtesting against historical data produces the most useful results. The main area of concern for me is that Sharpe ratio alone does not sufficiently account for overfitting. It may be that a balanced compromise can be found in moderating Sharpe ratio by (for instance) the volatility of returns. This is one of the reasons that I'm trying optimizations against some of the other Lean statistics using the ConfiguredFitness class that's now been provided inthe optimizer project. It may be that customized fitness measures could be the the arena for genuine innovation in this field.

I like the use of “multiple out of sample” periods. Do you have any idea of how to implement it? Thinking out loud, I imagine the following scenario: there are many periods of out of sample data (OOSD), say, a random month for each year in the optimization period). Then every time you run an OOD test, a month is selected randomly, or maybe many. Maybe in that way we can avoid (in part) the problem of systematic use of the OOAD, that ends up in more overfitting by incorporating the OOSD as in sample data, but in a second stage.

I know backtests are useful, the point is not to fool yourself.

Respect to the statisitcs and the fitness formula, I can’t agree more, is the truly art of the whole process... and the secret sauce ;)

Its been quite clearly established that the singular Sharpe ratio and naive in-sample backtesting are probably a pestilence at the level of an epidemic.

My personal position is that the accepted terms in use already precludes the possibility of a "holy grail". We’re drawn to such illusions by the bewitching flaws of language and the beguiling form of mathematical notation.

Either you can build firmer foundations on this unsteady ground, or you can dig deeper and discard the entire existing edifice.

Is % profit not a good measure? I understand drawdown etc, but with forward-testing you've implemented, this would be my first choice. Money is the point right?

Return can easily be misleading when an optimization fixates on a single rare event. This is less an issue when drawdown is taken into account.

Got it, thanks guys :-)

Any progress on running parallel backtest in the cloud?

I'm very new here and to trading in general, so please excuse my ignorance. Is this basically equivalent to MetaTrader4's ability to run a backtest with inputs set to a range (e.g. start, step, end) and run through each combination of that range to identify the settings with the best outcome? I was really hoping this was part of QC/Lean out of the box. Is running it on a local machine really that intensive that it should be done on a cloud instance instead? MT4's optimization didn't seem to be too slow as a desktop client. I know MT4 has a "genetic" optimization option that I haven't looked into yet. Does LeanOptimization not have a non-genetic optimization option? What am I missing?

James Candan, I'm kind of new to QuantConnect, but as far as I can tell, it's not too hard to run LEAN locally. The problem is that you'll have to constantly download new data if you want to be backtesting on current data.

Everyone, please correct me if I'm wrong.

^That's accurate. I've had to resort to doing some things locally (e.g. intraday charting) in cases where data is available for download.

Jared Broad you said `we're just finishing interactive brokers live trading and then will launch it :) ` back in March of 2015. Did this ever get pushed out? I'd love to be able to run a backtest with a range of variables and see which works the best.

of course, check the main website and scroll down

Michael Manus I meant the optimzation features, not the interactive brokers features... I don't see anything about that on the home page.

oh sorry I misunderstood you

 i think it got delayed because of the many problems of overfitting strategies and more urgent things

but it seems james is still working on it when i check the commits on the optimiztion project

Its a work in progress; we recently changed our cloud architecture to make this possible so we'll be adding it in 2018 -- please feel free to follow along with the dedicated project here. Our general motto is bugs before features so it has been delayed slightly.

Noted Jared. If that matters, my opinion would be to put this feature in top priority from a business perspective.

Otherwise, I have a question concerning the following statement on the page  https://www.quantconnect.com/lean/docs#topic14.html

 "Optimization / Parameterized Algorithms
Variables can be denoted as parameters and set via configuration files on launch. This opens up the possibility for optimization of algorithms and searching parameter sets."

Is this true? Can we actually run parameter optimizations in local? If yes, could you update this documentation page to explain the process in more details.

Thank you. Cordially,

Benjamin.

"Can we actually run parameter optimizations in local"

Ben, I've provided some information regarding this previously in this thread.

Thanks Jared. I am being a bit lost between the different docs (QC Online / LEAN), what the open source and staff developers do, and the forum threads that can extend on very long timelines ;).

To end on a piece of wisdom, as simple backtesting in itself cannot show any statistical significance, there are currently no tools available to properly test an algorithmic strategy in the crypto asset universe (such as MC analysis provided by TradeStation: http://help.tradestation.com/09_01/tswfo/topics/monte_carlo_wfo.htm). I am looking forward to QC and other players to navigate towards such functionalities, and particularly to QC since you already support realistic brokerage models.