Walk Forward Optimization - Can we have this too?

QuantConnect Community Discussions

QUANTCONNECT COMMUNITY

Join Our Discord Channel

Join QuantConnect's Discord server for real-time support, where a vibrant community of traders and developers awaits to help you with any of your QuantConnect needs.

Draft Discussions

Bookmarked Discussions

Walk Forward Optimization - Can we have this too?

Last comment by - March 2023

Walk Forward Optimization - Can we have this too?

Walk Forward Optimization - a process to avoid curve fitting in strategy development by segregating data into multiple segments.

Continue Reading

Share New Research

Start New Discussion Sign up

SEARCH DISCUSSIONS

366,300 Quants.

Become a Quant

Online Now

+961

Code of Conduct

VOTE FOR UPCOMING FEATURES

Share your input and vote on our future direction.

LEAN Roadmap

JOIN OUR Community MAILING LIST

Create an account on QuantConnect for the latest community delivered to your inbox.

Sign Up Today

Walk Forward Optimization - Can we have this too?

Uncle bob | uncle bob | February 2015

Walk Forward Optimization is an optimization process that addresses the issue of curve fitting in strategy development.

Walk Forward Optimization segregates the data series into multiple segments, and each segment is divided into an in-sample (IS) portion and an out-of-sample (OOS) portion.

Parameter optimization for the strategy is performed using the IS portion of the first segment. The same parameters are then used to back test the strategy on the OOS portion of the same segment. The process is repeated for the remaining segments.

The OOS performance results from each of the segments are considered "real" instead of "curve-fit" because the parameters that produced the OOS results were generated from IS data.

Author

February 2015

Upvote

3 people upvoted this

Jared Broad INVESTOR

QuantConnect | February 2015

It shouldn't be too hard once we're launched the optimization features. Will keep it in mind for the planning. If you have other feature requests please file them in github - https://github.com/QuantConnect/Lean/issues

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Reply

Upvote

2 people upvoted this

Mark Jonhston INVESTOR

April 2016

any updates on walk forward, or are there any optimization features releases? thanks

Reply

Upvote

1 person upvoted this

Non Compete INVESTOR

No Organization | February 2023

Sorry to bring up an old thread, but was walk forward optimization added? Is there way to make your own optimizer in python or only through contributing to the git repo? Thanks!

Reply

Upvote

Jared Broad INVESTOR

QuantConnect | March 2023

Officially scheduled for 2023. We're currently cranking through bugs but will hopefully get to features soon.

We're hiring for LEAN C#/C++ Engineer if anyone is interested in doing this full-time!

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Reply

Upvote

2 people upvoted this

Garrett Grow INVESTOR

March 2023

Wow! We're get walk-forward this year. I can't wait!

Reply

Upvote

1 person upvoted this

Nicholas Stein INVESTOR

January 2016

I recommend you go through the source code for QCAlgorithm.Trading and look carefully at the Liquidate and SetHoldings methods to make sure that they are doing what you intend. You may be a victim of network latency. How do you know that the Liquidation is happening before the Market order in SetHoldings is filled? I do not believe you can rely on IB to execute your orders in the same sequence as you entered them in your algorithm. For example, a the market buy may occur before the market sell happens. There is no way to know what is happening inside their order router and how fast things occur. The two orders may be being handled on different parts of their cluster, and on different threads almost for sure. You cannot assume all interactions with IB are atomic and you have to assume that they are asynchronous. With 45,000 share order liquidity can get also in your way. An order that size is almost certain to get the attention of the HFT algorithms and the market will move away from your market order and you will get partial fills at different levels. If you think that paper trading is bad, wait until you try it live.

Reply

Upvote

Jiaqi chen INVESTOR

January 2016

Thank you for your suggestions. The idea about order handling is very useful and I will definitely look into it.

Reply

Upvote

Jared Broad INVESTOR

QuantConnect | January 2016

Normally invalid orders have error messages attached, these may be in your algorithm logs. Did you see any detailed explanation?

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Reply

Upvote

Jiaqi chen INVESTOR

January 2016

Hi Jared, I have checked the log file. It started with 2016-01-13 12:01:01 New Order Event: Id:2 Symbol:XIV Quantity:-900 Status:PartiallyFilled and the final liquidation of XIV was at 12:01:06 as 016-01-13 12:01:06 : New Order Event: OrderId:2 Symbol:XIV Quantity:-341 Status:Filled Then there is no event ID3 for the VXX buy order although I see the 2016-01-13 12:01:02 SELL >> 21.335 2016-01-13 12:01:02 BUY >> 23.6 log output from my own code. In fact there was nothing between the Fill of orderId:2 and my stopping of the algo 2016-01-13 21:39:48 Algorithm Stopped There was no error message with the 3rd order in the log file although I can see the 3rd order as invalid in the orders tab. I guess I will play with some other order logic to check the executions. BTW, can you tell me why there are both Id:2 and OrderID:2 order events in the log file as shown below and why Quantitiy of 0 is submitted? Such messages showed up using either Liquidate method or SetHoldings. Maybe the message does not matter if I do not concern too much about the filling price for now? 2016-01-13 12:01:01 New Order Event: Id:2 Symbol:XIV Quantity:-1900 Status:PartiallyFilled 2016-01-13 12:01:01 New Order Event: Id:2 Symbol:XIV Quantity:-2000 Status:PartiallyFilled 2016-01-13 12:01:01 New Order Event: Id:2 Symbol:XIV Quantity:-700 Status:PartiallyFilled 2016-01-13 12:01:01 New Order Event: Id:2 Symbol:XIV Quantity:0 Status:Submitted 2016-01-13 12:01:01 New Order Event: OrderId:2 Symbol:XIV Quantity:-4200 Status:PartiallyFilled I will look further into the order mechanism to get a better understanding. Thanks and good night:-)

Reply

Upvote

Michael Handschuh INVESTOR

January 2016

Typically invalid orders will have the 'Tag' column populated in the orders tab with the reason why it was rejected.

Reply

Upvote

1 person upvoted this

Nicholas Stein INVESTOR

January 2016

Hey @jiaqi chen, I struggled with the async nature of the process for a long time and tried a bunch of different approaches. Now, in each OnData, I check for a signal. When I receive one, I place an order and put a lock on placing more orders for that symbol. In the OnOrderEvent, I check the orderEvent.OrderStatus. If the status is Filled or Cancelled, I unlock the symbol for more orders. If it is partially filled, I increment a TradeAttempts variable for the symbol. Strategy[symbol].IsActive is the lock. If IsActive == false, I do not enter another order. Strategy[symbol].TradeAttempts counts the number of partial fills or bars where nothing happened. Back in OnData, when TradeAttempts reaches 3 attempts, I cancel the order. When the cancellation comes through the OnOrderEvent, I unlock IsActive and reset TradeAttempts. Attached is a sample algorithm which uses this technique. Look in MultiSymbolStrategy.cs for the IsActive and TradeAttempts.

Reply

Upvote

1 person upvoted this

Jiaqi chen INVESTOR

January 2016

Thank you all for the useful information. It will take me some time to digest and it has been a great learning experience.

Reply

Upvote

Jiaqi chen INVESTOR

January 2016

HI Michael, this order is weird w/o any tag. The log file only show Time Symbol Price Quantity Type Status Value Tag 2016-01-12T15:01:00.000935Z XIV 21.84952008 45841 Market 3 1001603.85 2016-01-13T17:01:00.000944Z XIV 21.33149233 -45841 Market 3 -977856.94 2016-01-13T17:01:00.000944Z VXX 0 41353 Market 7 0 I guess I will simply start with Nicholas' suggestion and play with it again. Thanks.

Reply

Upvote

Jiaqi chen INVESTOR

January 2016

One further questions on the same issue. Now I am playing with only one simple logic to see how a simple market order could be executed. SetHoldings(Symbol, 1); In my live test this Friday morning it became Symbol Price Quantity Type Status Value XIV 0 45384 Market 1 0 and continued for multiple times. But when I checked my IB paper account the order was placed and in fact the market order was submitted and filled twice to use the all the margin to buy the ETN. I guess even if I am trying to get the order status as suggested I guess the 0 status would still make the love trading fail? Has anyone encountered such disconnected status between the broker and quantconnect order status?

Reply

Upvote

Jared Broad INVESTOR

QuantConnect | January 2016

Hi @Jiaqi, please use the latest "master" build of LEAN. We've updated it to send back more order status information.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Reply

Upvote

Jiaqi chen INVESTOR

January 2016

Hi Jared, I have switched to the master build during the weekend and it still has problems with my two days' experiment this week. The buy and sell market orders were carried out successfully on Monday but the market order was not able to flip the position this morning. I think it might be related to the size of the order and IB's fat finger setting. I will post my experience and my further experiments. Hopefully I can find the problem and benefit the great community here. The trade log shows a status of 1 for the first failed market order at 16:21:00.00084Z and it continued to issue market orders at my consolidated trade bars all with a status of 1. I have also checked my IB trade log, the market orders were submitted but not even partially filled. Then I try to flatten my holdings in IB using a market order but IB turned down my order saying the order is too large for some fat finger threshold (I do not remember the exact statement and I could only paraphrase what I saw). So instead of flattening 100% I flattened 50% apiece to fully liquidate the position. I have seen the fat finger message before in my previous failed experiments and I started to think it might be the cause of those status 1 orders shown both in quantconnect log and IB order log. I have reapplied the algo in quantconnect paper trading to see how it is doing for the next two days and if it is sucessful I will simply reset the order size to be 5000 shares in my IB paper trading account to see whether all the market orders will be executed. Meanwhile, if my thinking regarding the fat finger setting is correct I still have no clue why the orders were successfully filled on Monday as they have the same size (90000 shares) as my failed orders. Any further suggestion is appreciated. Thanks.

Reply

Upvote

Nik milev INVESTOR

January 2016

Not sure if this helps, but here is an observation about async nature of the orders in "live" IB trading. When you send an order, there are few seconds before the callabck triggers. So, if your algo comes back (OnData() executed) before the callback triggered , and there are no proper checks, it will attempt to send the order again. And this is what i experienced when working with 1 sec bar trading paper live on IB: when the buy/sell order is sent, it takes around 15 sec. to complete. So, the OnData() triggers several times (since bar is 1 sec in my case). Well, the flag Portfolio.Invested is still false (or true if i try to close position), and the algo attempts to buy (or close several time) several times and I get Reg T violation. Good luck Nik

Reply

Upvote

Jiaqi chen INVESTOR

February 2016

Hi Nik, Thanks for the information. I am only using 5 mins bar and the usual execution time is ~5-6 seconds if executed. So it is kind of weird for me to have similar issues as you have described. A quick update on the experiment. I have reduced the position size to be ~$200K/ trade for a security with about $600M daily volume. The # of shares is definitely below the fat finger threshold in IB when I played with TWS. The auto-trading was Ok yesterday for about 26 trades but the same status 1 issue happened again this morning for the opening trade while the trade was executed in the IB account (no error message in the order log file). Then the market orders kept piling up until the capital is used up in the paper trading account. I am totally lost at what to do next. I guess the orderstatus like var newTicket = MarketOrder (Symbol2, -1*holdings2, asynchronous:false); newTicket.Status ... might be of little use for this situation as the status is not inline with the actual status in IB. In other python/matlab IB interface we could check the actual position in your account, do we have the same feature? That is the only way I have been thinking about to deal with this weird situation or should I totally abandon MarketOrder and use limit order only? Is that just a problem with IB only? Thanks.

Reply

Upvote

Jared Broad INVESTOR

QuantConnect | February 2016

Hi Jiaqi, if you think there's an issue please send an email to https://www.quantconnect.com/support with your project and algorithm id and we can look into this for you. The synchronous market order has a timeout, if your order takes longer than the timeout(5 seconds) to execute it will pass through. You can set the timeout in Transactions class - https://github.com/QuantConnect/Lean/blob/master/Common/Securities/SecurityTransactionManager.cs#L113


Transactions.MarketOrderFillTimeout = TimeSpan.FromMinutes(30);

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Reply

Upvote

Nicholas Stein INVESTOR

February 2016

Jacki, As Jared said, you can set the timeout on market orders, but that the Ticket returned by the SecurityTransactionManager cannot be guaranteed to agree with what IB has on their books. Part of the reason is partial fills and part of the reason is network latency. Your code needs to handle both situations. The best way is to block your order sending routine from submitting further orders, in the line of your code before you send the order. You must implement OnOrderEvent to assure you get confirmation from IB on what happened on their end. The orderEvent object will give you the status and you react to that. The first time it fires, the status will be Submitted; even on a Market order. This event is a good thing because you can then tell that IB received the order and has submitted it to their router or whatever they use. On the second firing of OnOrderEvent, if the orderEvent's Status is PartiallyFilled, you do not want to remove your block because the order is still in the works. You only unblock your sender when the orderEvent Status is either Filled. That is how you know that IB has Filled the order. The same logic applies when you cancel an order. You then have to wait for confirmation from IB through the OnOrderEvent before you unblock your sender. PseudoCode: In your OnData. Block your sender Submit an order ...wait for confirmation of Fill or Cancel from IB In OnOrderEvent Unblock your sender when it arrives.

Reply

Upvote

Jiaqi chen INVESTOR

February 2016

Thanks guys. I will try to follow your ideas and hopefully it would work :-)

Reply

Upvote

Nik milev INVESTOR

February 2016

From your previous attempts: check the logs and find IDs of orders that you sent but apparently not executed. Check the time of these orders. Might be you are sending them correctly but IB is simply not executing those in a "reasonable" time. Also compare QC logs of the orders with TWS Order window. (Of course make sure you date/time on the machine you run "live" trading is set correctly). These timings might give you a hint what happened (times between orders, orders are seen in TWS, orders partially filled and etc.)

Reply

Upvote

Jiaqi chen INVESTOR

February 2016

Hi guys, Thanks for all the suggestions especially on using Transactions.MarketOrderFillTimeout = TimeSpan.FromMinutes(30); and Nicholas' sample code of handling different order types. With the new master lean engine and a setup of Transactions.MarketOrderFillTimeout = TimeSpan.FromMinutes(10); my naive paper trading test using 5 mins consolidated bars were successful during the weekdays. But it mysteriously continued to place an order on Saturday and I have received the following message: Runtime Error: Algorithm took longer than 10 minutes on a single time loop. Stack Trace: at QuantConnect.Isolator.ExecuteWithTimeLimit (TimeSpan timeSpan, System.Func`1 withinCustomLimits, System.Action codeBlock, Int64 memoryCap) [0x00000] in :0 at QuantConnect.Lean.Engine.Engine.Run (QuantConnect.Packets.AlgorithmNodePacket job, System.String assemblyPath) [0x00000] in :0 User: 9707, Project: 207541, Algorithm: L-6013a84ec2a556565750d0ce54bd0d8d I am doing nothing fancy but using Order(Symbol,total_qnt,OrderType.Market); to test the order placement In fact this is the second weekend I am see the same error. In the backtest the algo was able to identify the non-tradings days with a problem. I think I may be able to handle this problem by checking whether it is a weekday or more safely, the security.Exchange.ExchangeOpen signal. It still feels weird that I would see such errors during the non-trading days. Any idea how it happens and how to solve it? Thanks.

Reply

Upvote

Jared Broad INVESTOR

QuantConnect | February 2016

@jiaqi chen; correct you shouldn't be able to place an order on Saturday. What code is surrounding / triggering the market order? You might be placing a market order in after market / weekend times which is then converted to a "Market On Open" order.

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Reply

Upvote

Jiaqi chen INVESTOR

February 2016

Hi Jared, I am testing nothing fancy but a simple market order of Order(Symbol,total_qnt,OrderType.Market); on every 5 mins consolidated bars with a minute resolution from AddSecurity(SecurityType.Equity, Symbol, Resolution.Minute); Transactions.MarketOrderFillTimeout = TimeSpan.FromMinutes(10);. Based on what I have seen from the backtest logs I think the market order is placed 1 min after the trigger signal thus if the signal is triggered at 4:00pm EST the algo will place a market order at 9:31am the next day if it is trading day. If I am using a Resolution.Second then the market order will be placed at 9:30:01am the next day if it is trading day. Did I get that part right? If I got this market order placement concept wrong then I can definitely add a time constraint to not fire the order if it is within 3 mins of market close. I use to use setHoldings(xx,xx,1) to flip the positions and I switched to the current cumbersome codes suspecting the setHoldings() function was giving me troubles. I have attached the project for your reference and thank you for your suggestions.

Reply

Upvote