How to compare backtest variations - API?

Interesting. I've used excel spreadsheet to gather and compare outputs of my algo with various parameters (momentum period, resolution, risk threshold). I'm sure there is better way of doing this, but had to postpone this topic alltogether.

Pawel Kowalke are you just recording the changes & results manually?

Jared Broad I understand. Do you have insight as to the API that has been mentioned on the forums? Is it still around?

Jean - any optimization, manually or api/otherwise, is not supported for now. It's just not sustainable and creates massive load on the system causing deterioration of service for everyone. We are almost ready with dedicated servers for people to do this type of testing.

Just as a quick survey: If there was an optimization feature available, would you be comfortable covering the computing costs? (i.e. direct pass-through of AWS rates). Running 500 backtests for 1 hour would probably cost ballpark $50+. 

Jean Lepisto I do just that. It would be interesting and useful at some point, to automate that. Just think of possibilites such as using the results of backtests within algorithm! :)

Jared Broad I think its fair to cover our own cost. Whether that is an instance paid for monthly similar to the paid live server (ex: paid private backtest server) or covering the costs of the runs.

What I personally would prefer is a predictable rate. So whether it is a flat fee or a calculated estimate once a build has been completed, or something similar so costs don't surprise me and add up.

My preference would be to have a flat monthly rate and be subject to the limits of the instance capacity. So if I can only handle 20 concurrent backtests, or it takes 2x as long then so be it, and I can upgrade/downgrade the instance based on my performance needs at a given time.

That being said, would I pay $50 to start to optimize an algo? Sure, when I'm reasonably confident that I am somewhat close to finalizing. If it runs for 10 hours would I pay $500? Probably not until I had proved it further (paper trading or otherwise)

From my perspective as the algo is developed and backtested that's where the most amount of time, effort and likely server resources are consumed. Especially for us noobs. We have a system that worked somewhere and are developing it while learning code. 

In my case, we have gone through a number of iterations not only to debug but also to try to emulate the strategy performance from visual backtesting tools (ex: optionstack) because these tools aren't completely transparent in how they're processing the strategy, slippage, fills, etc. 

So this means lots of backtest runs for debug and trying to reverse engineer the exact details. I'm sure that means us newbies are using up much more than others. Having a paid instance to do this faster and without affecting others would be very fair in my opion.

Another benefit would be faster processing for longer backtests. A few days or a month is quick, but multiple years can take a lot longer. I find it invaluable to test long term across many market conditions instead of just snapshots of selected timeframes/conditions.

Further, I'd gladly pay for a feature that would allow for import of strategy from a tool like Optionstack vs struggling with the code even with a programmer's help. That to me and many other non-programmers would be gold.

Perhaps you got more response than you asked for, but I appreciate you asking and reading.

Thank you Jean for the detailed answer -- appreciate it. We'll do our best to factor these concerns into the design.

As a note on this. My strong prefernce would be to have Optimization run using something like spot instances... Rather than dedicated VM time.

Even with spot instances Chris it will cost "tens of dollars" per optimization. If people are willing to cover it we can implement this feature but we're not sure of the demand at this time. 

Jared Broad "tens of dollars" to verify you have the best performing algo out of the variables you can change sounds like a great deal to me. I don't know why anyone wouldn't want to either confirm they had the best performance for their strategy for that kind of $.

To be fair, I trade options so there's a lot more variables than just stocks for example, and that may be influencing my opinion disproportionately and the additional factors may require more resources than tens of dollars would provide?

Hi Jean,

We appreciate your suggestion, and will take it into consideration. Thanks for your support!