Share New Research 
HI guys,
I ran some back tests and live trading (IB) for the same ago and observed differences. I would like to make the back testing more realistic. Here is what happens:
Assume I run on 1 sec resolution and current price of the "abc" security is $10.00 and I have custom slippage e.g. 0.03%
Backtesting: when i send limit order to buy at $10.05, the fill price is $10.00 - it seems that it takes current closing price, since it is within the buy limit.
Live run: it fills at e.g. $10.04
Is it possible in backtesting to change the orders fills to happen on the next bar, e.g. to set this somehow during initialization ? This will make it a bit more realistic.
Other ideas a very welcome!
Thanks
Nik
Stefano Raggi
Hi Nik,
just a couple of notes to clarify since you mentioned limit orders:
This is the logic for limit buy order fills in this model:
if (prices.Low < order.LimitPrice) { //Set order fill: fill.Status = OrderStatus.Filled; // fill at the worse price this bar or the limit price, this allows far out of the money limits // to be executed properly fill.FillPrice = Math.Min(prices.High, order.LimitPrice); }from your example it seems the High of the next bar ($10.00) was actually lower than the order limit price ($10.05).
A few suggestions:
Nik milev
Thanks for the quick response!
So, since the limit order fills on next bar, you think next bar is $10.
How about when during next bar/second there was no trade?
I would guess that the previous bar close of $10.00 would be carried over
and will cause the fill to happen on next bar which is in effect unchanged
previous bar, due to no trades.
What do you think?
As you said may be using tick is better option
Thanks
Nik
Stefano Raggi
Hi Nik,
the limit order fill is not guaranteed to happen on the next bar, depending on the price action it could happen later:
e.g. if a limit buy order is at $10 and the Low of the next bar is at $10.05 there will be no fill and the order will remain active.
If there are no trades in the resolution period, the bar will be copied forward but should not be used for fills, that would be a huge LEAN bug :)
Nik milev
Thanks for the clarification that the artificially copied bar will not be
used for fills, I did not know this!
I will try to create a clean test to better see what happens.
Thanks for the support!
Nik
Nik milev
Hi Stefano,
I cleaned a little bit the test and ran it again. This time I used TYD and DRN - both trade very few shares. Basically I buy every 15th minune and sell every 33rd minute (sending limit orders). If after 1 minute the orders are not filled, they are canceled and all is liquidated.
I put limit order to enter few cents above the current price and to exit few cents above the current price.
As I see it, it alwas enters and almost never exits - check TYD. So it seems that it uses next bar to enter, even if there is no trade.
Unless I have a mistake in the code which I cant see, since it is hard to debug, it seems to me that e.g. the algo uses next bar of TYD to enter and caries over TYD bar to next second - prices and volume seem suspiciously the same from 10:15 till 11:30 and etc.
Your opinion is greately appreciated.
Thanks
Nik
9/17/2018 10:15:00 AM BUY order1 sent, .status=New, limit= 40.090, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 10:15:00 AM BUY order2 sent, .status=New, limit= 23.430, curPrice=23.410, high=23.410,low=23.410, vol=300.000
9/17/2018 10:15:00 AM, Will print next bar
9/17/2018 10:15:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 10:15:01 AM, barPairNow[1]=23.410, high=23.410, low=23.410, vol=300.000
9/17/2018 10:15:01 AM Entry Orders Executed , order1.FilledPrice=40.07, order2.FilledProce=23.41
9/17/2018 10:33:00 AM EXIT order1 sent, .status=New, limit= 40.080, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 10:33:00 AM EXIT order2 sent, .status=New, limit= 23.350, curPrice=23.360, high=23.360,low=23.360, vol=500.000
9/17/2018 10:33:00 AM Exit orders sent
9/17/2018 10:33:00 AM, Will print next bar
9/17/2018 10:33:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 10:33:01 AM, barPairNow[1]=23.360, high=23.360, low=23.360, vol=500.000
9/17/2018 10:33:01 AM exit order1 canceled
9/17/2018 10:33:02 AM HandleState_PendingOrderExitCancelationSent - Liquidated ALL exit orders if any
9/17/2018 11:15:00 AM BUY order1 sent, .status=New, limit= 40.090, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 11:15:00 AM BUY order2 sent, .status=New, limit= 23.170, curPrice=23.150, high=23.150,low=23.150, vol=5.000
9/17/2018 11:15:00 AM, Will print next bar
9/17/2018 11:15:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 11:15:01 AM, barPairNow[1]=23.150, high=23.150, low=23.150, vol=5.000
9/17/2018 11:15:01 AM Entry Orders Executed , order1.FilledPrice=40.07, order2.FilledProce=23.15
9/17/2018 11:33:00 AM EXIT order1 sent, .status=New, limit= 40.080, curPrice=40.070, high=40.070,low=40.070, vol=200.000
9/17/2018 11:33:00 AM EXIT order2 sent, .status=New, limit= 23.180, curPrice=23.190, high=23.190,low=23.190, vol=406.000
9/17/2018 11:33:00 AM Exit orders sent
9/17/2018 11:33:00 AM, Will print next bar
9/17/2018 11:33:01 AM, barPairNow[0]=40.070, high=40.070, low=40.070, vol=200.000
9/17/2018 11:33:01 AM, barPairNow[1]=23.190, high=23.190, low=23.190, vol=406.000
9/17/2018 11:33:01 AM exit order1 canceled
9/17/2018 11:33:02 AM HandleState_PendingOrderExitCancelationSent - Liquidated ALL exit orders if any
Stefano Raggi
Hi Nik,
I checked the data files for TYD, these are the first two (second resolution) bars in the data file for 9/17:
34200000,400700,400700,400700,400700,200 // 09:30:00
51033000,399800,399800,399800,399800,200 // 14:10:33
Since fill forward is enabled by default, you will receive the first bar in OnData multiple times (from 9:30 until 14:10 when there was a new trade), this explains why you are seeing the same prices (and volume) repeated in the logs.
Regarding the limit order fills in backtesting, while it is true that limit order fills are always filled on the next bar, the prices used to determine if a fill occurred are from the last bar received (which can actually be a previous bar copied forward -- apologies for the confusion I might have caused in my previous post).
In equity backtesting we don't have bid/ask quotes, so we can only use bars created from trade ticks.
After running your test algorithm locally, I verified that the output is expected, this is the sequence of events:
at 9:30 the first bar comes in with all OHLC values equal to 40.07
at 10:15:00 the buy order for TYD was submitted (limit price = 40.09)
at 10:15:01 the buy order was filled at 40.07
- the fill price was lower because in the current bar 40.07 is the highest value traded
at 10:33:00 the sell order was submitted (limit price = 40.08)
- this order was not filled because the limit price was higher than the High of current bar (40.07)
Sadly when the first order was submitted at 10:15 the only price data available was a bar from 45 minutes earlier.
In live mode you would get real time quote prices, so differences in fill prices are expected.
In this case, even using Tick resolution wouldn't help, there were no trades from 9:30 to 14:10.
Returning to the original question on how to make backtesting more realistic, I think maybe a custom fill model could try to estimate the bid/ask prices and handle limit order fills more accurately (modeling thinly traded equities like TYD might be hard though).
Jared Broad
Thanks for raising this case Nik; its an interesting extreme. We'll review the fill models and make sure they'll behave as expected - filling on the next appropriate bar open/high/low/close price based on the order type.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Nik milev
Thanks for looking at this issue!
Filling on last and carried over old bar creates problem and skews test
especially for not so traded securities, and when we use higher resolutions
( e.g. seconds).
Few things come to my mind I would like to share with you:
1. It would be beneficial for me as a developer to somwhow know if current
bar is built from real data or just carried over.
There are several ways to do this, and one way would be to add a property
to TradeBar, showing the time of the last ticker/trade. This way, I can see
that the closing price is not in e.g. minute/second (my resolution) and
will act accordingly.
2. Another thing to do along with above idea (to notify the algo that the
bar is artifficial) is to provide several fill modes.
In case when no bar/trade is in the next resolution interval, an
approximation artifficial bar can be created, based on the last trade and
future trade. This bar prices will be approximation - the close time is to
the future trade, the better its prices match.
I guess same can be done for volume (or always set volume=0) for
artifficial bar.
Currently, trading seconds resolution for average volume trading securities
artificially boost the returns.
I am looking forward to know more about your decision and implementation!
Thanks
Nik
Stefano Raggi
Hi Nik,
we totally agree that order fills for limit (and stop/stoplimit) orders should never be emitted if there is no new data,
there is a proposed fix here (currently under review):
For the volume in fill-forward bars being copied over from previous bars, this was definitely a bug and has already been fixed and merged yesterday:
Regarding your two questions:
1. In OnData, to know if a bar is real or fill-forward, you can use the BaseData.IsFillForward property:
var isFillForward = bar.IsFillForward;2. There is no easy way to generate artificial bars in LEAN for any symbol, all existing models should work with the actual (real) trade data.
Hopefully the two bug fixes above will improve the accuracy of your backtesting results.
Nik milev
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!