Precise Market Open and Close Times for Daily Data

This is great! Does this also mean that for backtesting, daily bars will emit at the time of close, or will they still emit at midnight?

This is a great update. Is there a way (other than using a custom calendar function) to take advantage of this with bar consolidators for equities? The goal would be that they also fire at the day's closing time rather than at midnight.

Hi everyone,

From Lean Engine version 16570 onwards, precise market open and close times are the default for daily data.

If your algorithm uses daily resolution, you may notice the following changes:

- Auxiliary events like Dividends and Splits, which previously arrived at midnight with price data, will no longer include price data in the slice. Ensure price data is available for a symbol by using 'slice.bars.contains_key(symbol)', more information.

- In backtesting, daily data used to be emitted at midnight alongside security changes. Now, the first price data is emitted at the exact close time (e.g., 4 PM for equities). As a result, submitting orders on security changes will fail since assets won't have a price yet. To address this, use the warm up feature or the price seeder during security initialization. We have create an issue so that price seeding is enabled by default.

- Using daily resolution with the consolidate algorithm API will now also respect precise market open and close times by default.

To revert to the legacy daily bars behavior, set 'self.settings.daily_precise_end_time = False' in the initialize method.

Thank you.

This is horrible. I see 100% difference in annual returns using daily data. Have I been relying on completely erroneous backtests?!

How can a change in bar close time create a 100% difference in performance when using daily data?!

What good is using self.settings.daily_precise_end_time = False if it produces great backtest results but doesn't match how live trading works?!

What a mess!

does this accurate daily end-times implemented for every dataset?

I found that

CBOE VIX doesn't provide daily end time for the data quote

i.e. 

        self.settings.daily_precise_end_time = True
       
         vix = self.add_data(CBOE, 'VIX', Resolution.DAILY).symbol

        history = self.history(CBOE, vix, 5, Resolution.DAILY)

        self.Log(history)

This return no timestamp  

                                             close  datasourceid   high    low   open  value
symbol      time                                                       
VIX.CBOE 2S 2024-08-31  15.00          2001  16.04  14.78  15.67  15.00
                       2024-09-03  15.55          2001  15.99  15.48  15.87  15.55
                       2024-09-04  20.72          2001  21.99  15.71  15.76  20.72

and indeed, even if i set the daily_precise_end_time = False, the result is the same. 

I am wondering if daily precise end time is not implemented for all dataset. Please help to clarify.
 

“Does this accurate daily end-times implemented for every dataset?”

Good question - no this is only applied on data covered in our market hours database (equities, index, futures etc). Custom data like CBOE does not have market hours. 

“Precise end-times changes my backtest result”

The most common cause of this would be intraday scheduled events, while using daily data. My guess would be depending on the time of the event it would now be triggered before or after the day. We always recommend using intraday data (hour-minute) for scheduling intraday events as its the best way to ensure accurate backtests.

Note, attempting to run the solution in the tutorial

https://www.quantconnect.com/learning/task/158/Preparing-Indicators-with-History

It directs me to this page with an error https://qnt.co/3YHaWHL 

5 | 3:55:23 : Backtest deployed in 7.474 seconds
6 | 3:55:27 : Accurate daily end-times now enabled by default. See more at https://qnt.co/3YHaWHL. To disable it and use legacy daily bars set self.settings.daily_precise_end_time = False.
7 | 3:55:27 : Backtest Handled Error: AMD R735QTJ8XC9X: The security does not have an accurate price as it has not yet received a bar of data. Before placing a trade (or using SetHoldings) warm up your algorithm with SetWarmup, or use slice.Contains(symbol) to confirm the Slice object has price before using the data. Data does not necessarily all arrive at the same time so your algorithm should confirm the data is ready before using it. In live trading this can mean you do not have an active subscription to the asset class you're trying to trade. If using custom data make sure you've set the 'Value' property.

The tutorials only returns results if I disable the new features as described here `self.settings.daily_precise_end_time = True`

 def Initialize(self):
        self.set_start_date(2019, 1, 7)
        self.set_end_date(2019, 4, 1)
        self.set_cash(100000)
        self.universe_settings.resolution = Resolution.DAILY
        self.settings.daily_precise_end_time = True
        self.add_universe(self.coarse_selection_function) 
        self.averages = { }

sorry, I meant by setting it “False” above

self.settings.daily_precise_end_time = False