Share New Research 
Hi,
According to documentation, History(symbol, periods, resolution), and all variations based on “periods”, should return a number of bars specified as “periods”. Here is a doc string:
“Gets the historical data for the specified symbol. The exact number of bars will be returned.”
But this is not the case. The code rather computes a start time, starting from a targeted end time, and rewinding time the specified amount of bars. So for instance, if you request 1000 second bars, it will go back 1000 seconds from the current time. But that does not guarantee you will get 1000 second bars. It will only be the case if the asset traded at least once every second over that 1000 seconds period. In other words, it will only be the case if there is a second bar for each single second over the 1000 seconds period, which can easily not be the case.
The code would rather need to simply request 1000 second bars from the given end time (going in the past) in the DB.
Here is an example showing the problem. The example will only work as long as the DB stays as-is. Right now, SPY data ends on June 13th in the DB but we are June 15th. If the DB gets updated, the example can execute differently and return a different number of bars.
class HyperActiveOrangePigeon(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2022, 6, 14) # Set Start Date
self.SetCash(100000) # Set Strategy Cash
symbol = self.AddEquity("SPY", Resolution.Second, fillDataForward=False, extendedMarketHours=True).Symbol
h = self.History(symbol, periods=1_000, resolution=Resolution.Second)
self.Debug(h)
self.Debug(len(h))
def OnData(self, data: Slice):
passAnd here is the log:
2022-06-14 00:00:00 : askclose askhigh asklow askopen asksize bidclose bidhigh bidlow bidopen bidsize close high low open volume symbol time SPY R735QTJ8XC9X 2022-06-13 19:43:22 375.40 375.40 375.40 375.40 200.0 375.36 375.36 375.36 375.36 100.0 375.360 375.360 375.36 375.36 25.0 2022-06-13 19:43:34 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.380 375.380 375.38 375.38 138.0 2022-06-13 19:43:41 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.360 375.360 375.36 375.36 70.0 2022-06-13 19:43:57 375.40 375.40 375.40 375.40 200.0 375.32 375.36 375.32 375.36 500.0 375.360 375.360 375.36 375.36 100.0 2022-06-13 19:43:59 375.42 375.42 375.40 375.40 500.0 375.30 375.40 375.30 375.32 500.0 375.400 375.400 375.38 375.38 5000.0 2022-06-13 19:44:08 375.39 375.40 375.39 375.39 500.0 375.29 375.29 375.29 375.29 500.0 375.400 375.400 375.40 375.40 14.0 2022-06-13 19:44:22 375.41 375.41 375.41 375.41 500.0 375.31 375.31 375.31 375.31 300.0 375.310 375.360 375.31 375.36 100.0 2022-06-13 19:44:34 375.39 375.41 375.38 375.41 1000.0 375.31 375.31 375.31 375.31 300.0 375.360 375.360 375.36 375.36 13.0 2022-06-13 19:44:59 375.41 375.41 375.41 375.41 500.0 375.31 375.32 375.31 375.32 1300.0 375.400 375.400 375.40 375.40 5.0 2022-06-13 19:45:00 375.39 375.41 375.39 375.41 500.0 375.31 375.32 375.31 375.31 1300.0 375.390 375.390 375.39 375.39 221.0 2022-06-13 19:45:31 375.46 375.46 375.46 375.46 500.0 375.37 375.37 375.35 375.37 500.0 375.370 375.400 375.37 375.40 200.0 2022-06-13 19:45:34 375.46 375.47 375.46 375.46 500.0 375.37 375.46 375.37 375.37 500.0 375.460 375.460 375.46 375.46 300.0 2022-06-13 19:45:39 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.370 375.410 375.37 375.41 42.0 2022-06-13 19:45:40 375.40 375.46 375.40 375.46 100.0 375.36 375.36 375.36 375.36 500.0 375.370 375.370 375.37 375.37 3.0 2022-06-13 19:45:42 375.44 375.44 375.40 375.40 1000.0 375.41 375.41 375.35 375.35 300.0 375.400 375.400 375.40 375.40 101.0 2022-06-13 19:45:45 375.42 375.44 375.42 375.44 500.0 375.35 375.41 375.35 375.41 500.0 375.410 375.410 375.41 375.41 301.0 2022-06-13 19:45:59 375.45 375.45 375.40 375.40 500.0 375.35 375.35 375.35 375.35 500.0 375.400 375.400 375.40 375.40 1.0 2022-06-13 19:46:08 375.49 375.49 375.47 375.47 1000.0 375.40 375.40 375.35 375.35 500.0 375.400 375.400 375.40 375.40 401.0 2022-06-13 19:46:25 375.52 375.52 375.52 375.52 1000.0 375.42 375.42 375.42 375.42 500.0 375.420 375.420 375.42 375.42 102.0 2022-06-13 19:46:28 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.450 375.450 375.45 375.45 36.0 2022-06-13 19:46:39 375.52 375.52 375.51 375.52 1000.0 375.50 375.50 375.50 375.50 200.0 375.500 375.500 375.50 375.50 9.0 2022-06-13 19:46:44 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.500 375.500 375.50 375.50 25.0 2022-06-13 19:46:45 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.520 375.520 375.52 375.52 133.0 2022-06-13 19:46:46 375.53 375.53 375.52 375.52 1000.0 375.50 375.50 375.50 375.50 200.0 375.520 375.520 375.52 375.52 500.0 2022-06-13 19:46:47 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.520 375.520 375.52 375.52 2.0 2022-06-13 19:46:48 375.53 375.53 375.53 375.53 1000.0 375.42 375.50 375.42 375.50 500.0 375.500 375.500 375.50 375.50 267.0 2022-06-13 19:46:55 375.53 375.53 375.53 375.53 1000.0 375.42 375.42 375.42 375.42 500.0 375.530 375.530 375.53 375.53 204.0 2022-06-13 19:47:24 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.590 375.590 375.55 375.55 5.0 2022-06-13 19:47:42 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.520 375.520 375.52 375.52 2.0 2022-06-13 19:47:57 375.62 375.62 375.60 375.60 500.0 375.52 375.53 375.50 375.50 500.0 375.570 375.570 375.57 375.57 517.0 2022-06-13 19:48:01 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.570 375.570 375.57 375.57 7.0 2022-06-13 19:48:07 375.61 375.62 375.59 375.62 500.0 375.52 375.53 375.52 375.52 500.0 375.550 375.550 375.55 375.55 48.0 2022-06-13 19:48:09 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.600 375.600 375.60 375.60 4.0 2022-06-13 19:48:18 375.59 375.59 375.59 375.59 500.0 375.50 375.50 375.50 375.50 1000.0 375.570 375.570 375.57 375.57 3.0 2022-06-13 19:48:22 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.590 375.590 375.59 375.59 40.0 2022-06-13 19:48:32 375.59 375.59 375.59 375.59 1000.0 375.50 375.52 375.50 375.52 500.0 375.550 375.550 375.55 375.55 7.0 2022-06-13 19:48:34 375.56 375.57 375.56 375.57 500.0 375.50 375.50 375.50 375.50 500.0 375.550 375.550 375.55 375.55 18.0 2022-06-13 19:48:36 375.57 375.57 375.56 375.56 500.0 375.50 375.50 375.50 375.50 500.0 375.520 375.520 375.52 375.52 25.0 2022-06-13 19:48:38 375.51 375.57 375.51 375.57 500.0 375.43 375.50 375.41 375.50 500.0 375.500 375.520 375.50 375.52 520.0 2022-06-13 19:48:49 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.480 375.480 375.48 375.48 30.0 2022-06-13 19:48:52 375.57 375.57 375.57 375.57 500.0 375.49 375.49 375.48 375.48 500.0 375.500 375.500 375.50 375.50 40.0 2022-06-13 19:48:54 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.510 375.510 375.51 375.51 2.0 2022-06-13 19:48:58 375.59 375.59 375.57 375.57 500.0 375.50 375.51 375.49 375.49 1000.0 375.510 375.510 375.51 375.51 73.0 2022-06-13 19:49:03 375.57 375.59 375.56 375.59 500.0 375.50 375.50 375.47 375.50 500.0 375.510 375.510 375.51 375.51 50.0 2022-06-13 19:49:07 375.59 375.59 375.59 375.59 500.0 375.50 375.50 375.50 375.50 500.0 375.500 375.500 375.50 375.50 25.0 2022-06-13 19:49:09 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN 375.510 375.510 375.51 375.51 26.0 2022-06-13 19:49:10 NaN NaN
2022-06-14 00:00:00 : 151Thanks for reading,
Fred
Louis Szeto
Hi Fred
It is the intended behavior. We calculate the lookback period based on the expected number of bars. The history won't return the required number of rows if there is no bar at the beginning of the period due to liquidity. This could ensure the functional integrity of History call for multiple Symbols.
Best
Louis
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Fred Painchaud
Hi Louis,
Thank you for your answer.
However, I don't understand your point wrt “The history won't return the required number of rows if there is no bar at the beginning of the period due to liquidity”. My original comment is not about the beginning of the period having a bar or not. It is about the fact that according to the documentation, the “periods” parameter is not an expected number of bars that you want returned but an exact, requested number of bars that you want returned. For instance, you request “1000 bars” to be returned, not “a maximum of 1000 bars” (which would correspond to an "expected number"). Also, the original comment is about the fact that all along the calculated lookback period, there can be missing bars, not only at the beginning.
For instance, the documentation for History(symbols, periods, resolution) reads:
Gets the historical data for the specified symbols. The exact number of bars will be returned for each symbol. This may result in some data start earlier/later than others due to when various exchanges are open. The symbols must exist in the Securities collection.
“The exact number of bars will be returned for each symbol.” - This is not the case.
“This may result in some data start earlier/later than others due to when various exchanges are open.” - Yes, this is the case, but not because the method seeks for existing bars but simply because it looks at when the exchange is open.
The documentation on the “periods” reads:
periodsThe number of bars to requestThe same is true about the documentation on the single symbol variation of History(symbol, periods, resolution):
Gets the historical data for the specified symbol. The exact number of bars will be returned. The symbol must exist in the Securities collection.
So as-is, those implementations of History are not designed to return a requested number of bars. They are designed to lookback “periods” amount of open time, according to market regular/extended hours. It is the only difference between those and the others with a start-end time and a span. Start-end and span are purely calendar time based while “periods” are exchange hours based. But the “periods” are not “bar based”, as the documentation implies.
Finally, according to code documentation again, the method GetStartTimeAlgoTz, in HistoryRequestFactory.cs, used by the “periods” variants of History, the intent is to “get the start time required for the specified bar count in terms of the algorithm's time zone”:
But as you know, to do that, the code must not look at open hours but directly at the available data. But that capacity is not implemented, History requests are all based on a start and end time, not on a number of bars to return. I believe the original intent was to find the start time that would yield a given number of bars, but it cannot work if you do not look at available data as you rewind time wrt open hours. And if you do look at available data, then you do not need to look at open hours.
My suggestion would be to either change the documentation or change the implementation. I would like to have an implementation of History that does return a given number of bars, irrespective of time (calendar or market). So it would be nice I believe to change the implementation so it matches the current documentation.
Hope this clarifies.
Cheers,
Fred
Fred Painchaud
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!