Futures data not accessible in research environment

Hi Tomasz Mierzejewski, the kernel dies when we try to access too much data.
The resolution of the data is minute so we have about 86400 data rows (closed market periods are not included) per day. I would advice you to make short-period requests, up to 5 days, then you can extract the data you need (maybe you just need a daily average) and move to the next date/time window.

I tried to get different resolution than Minute, but that doesn't seem to work. Do futures support Hourly/Daily resolutions? 

Not, yet. Only minute or higher. Please checkout the futures data here.
With pandas, you can easily compute hour or daily bars from the minute data.
That is the only way to do at the moment.

I used the exact example in the documentation. I get NO data at all. 

BTW, is natural gas futures on NYMEX symbol NG? Just checking.

thanks!

=========================================

es = qb.AddFuture("ES")
es.SetFilter(timedelta(0), timedelta(180))

# request the SP500 futures history from 01/11/2017 10:10 to 01/12/2017 12:10
future_history = qb.GetFutureHistory(es.Symbol, datetime(2017, 1, 11, 10, 10), datetime(2017, 1, 12, 12, 10))

# Get all futures data as a dataframe
future_history.GetAllData()

Future data in research is a known issue. Thanks in future please send bug reports to support@quantconnect.com 

Hi all!

We are happy to announce that the following code now works correctly in the new research!

Hi Martin,

Tested with ES and NG, CL etc.   kernel dies often but not always:  

Kernel Restarting
The kernel appears to have died. It will restart automatically.

Performance quickly deteriorates with increasing number of given months.  

I think there is a memory limit and if you hit the memory limit the kernel will die

HI everyone 

are the hourly and daily resolutions still not available within either the research or back-testing environment ?

secondly , can I access these series in continuous form as per the convention within this type of asset classes i.e. (adding the chains together without gaps , many methods are available and which one do you apply) ?

best regards

Hi Ashraf

Our data provider has provided minute, second and tick data only, you could easy resample the data by df.resample() method. As per continuous historical data, we support continuous future in research environment also:

qb = QuantBook()

# Canonical symbol
es = qb.AddFuture(Futures.Indices.SP500EMini).Symbol

# continuous future
history = qb.History(es, start_time, end_time, Resolution.Minute)
history = history.droplevel([0, 1], axis=0)  # drop expiry and symbol index, useless in continuous future

# resample daily data
df = history.resample("D").agg({'askopen': 'first', 
                                 'askhigh': 'max', 
                                 'asklow': 'min', 
                                 'askclose': 'last',
                                 'asksize': 'sum',
                                 'bidopen': 'first', 
                                 'bidhigh': 'max', 
                                 'bidlow': 'min', 
                                 'bidclose': 'last',
                                 'bidsize': 'sum',
                                 'open': 'first', 
                                 'high': 'max', 
                                 'low': 'min', 
                                 'close': 'last',
                                 'volume': 'sum'})
history

Best
Louis

hi louis 

your code is informative and steps are easy to follow however , in your experience is there a maximum limit that I can get say on tick data , or minute data resolutions within history ? 

your input is highly appreciated ,

secondly is there a way (paid or not) to get access to data on daily shorting costs( dollar wise )within the QC platform for each shortable instruments within the QC universe ?

your input is highly appreciated 

Hi Ashraf

That would be depending on your research node's RAM size, normally we'd expect 5MB consumption per security. You might upgrade your R-node in the pricing page.

For “shorting cost”, I think you meant the borrowing cost? Sorry we do not offer that type of data. In principle, that would be dependent on the broker you're using but not on the stock itself. Also, if the stock is “difficult” to borrow on that day, likely your additional order would make that cost even higher, so any historical data of that could only have little meaningfulness.

Best
Louis

thanks Louis am actually actively looking in to upgrading , yet I wanna make sure that I have best value for such monthly investment , however , due to the fact that am interested in expressing certain views via the use of certain type of futures called “dividend futures” , is there a way to access these on the platform , at least for the S&P 500 equity market ? by the way your code above worked just fine for me but in general is there a way to track when the 5 MB limit per security is used and how it changes in function of the history and data points accessed per each security ?

lastly , is it always possible to import data in to the platform via our account on IB ? if yes , then would you provide / code in some examples to see how that works 

looking forward to your response 

best regards

one more issue Louis I have run in to which is basically how do I add the additional time of day constraint to the below rule , basically i want to get in to 10 days before market close on the below dates 

if not self.Portfolio.Invested and self.Time.day==4 and self.DateRules.MonthStart():

if not self.Portfolio.Invested and self.Time.day==4 and self.DateRules.MonthStart():
   self.SetHoldings("SPY", 1) 
else:
  self.Liquidate("SPY")

Hi Ashraf 

Dividend futures are not available yet, but we'll certainly look into expanding our supporting assets in the future.

5MB is not a limit per security data, but rather an estimation of the size of that data, so you could purchase the node size according to your needs. Like 1500-stock research would most probably need an 8GB RAM research node addon.

IB data would need to be imported as custom data. Right now, the IBDownloader in LEAN's Toolbox was removed temporarily, so you will have to download and save the file in your own way. This is not affected in live trading although.

For 10 days after a position opening, maybe you could set up a dictionary to hold the day count and increase that in OnEndOfDay(Symbol) method:

def Initialize(self):
    self.day_count = {}
    
def OnData(self, data):
    ...
    self.SetHolding(symbol, 0.1)
    self.day_count[symbol] = 0
    ...
    self.Liquidself.day_count[symbol]ate(symbol)
    del self.day_count[symbol]    # delete the key-value pair in the dict
    
def OnEndOfDay(self, symbol):
    if symbol in day_count:
        self.day_count[symbol] += 1

Best
Louis

thanks Louis for the above info , so shall I expect for Like 3000-stock research + their associated options contracts would most probably need 16 GB RAM research node add-on ? kindly confirm if this is the case 

with historical periods covering crash of 2007-2008

lastly returning to the most pressing short-term need in regards to deploying an algo live for only two days per month on 5-10 tickers , what size do I need in terms of live trading node ?

and lastly is there a way to define the following within the code 

1- On.schedule method to define time rules for entry and exit in separate way 

2- How to add “ marketable limit order ” I have checked the order types but not what I want to implement , basically if current buy say at 10.50 then my buy marketable limit order will be between 10.50 up till say 10.55 but not anything further than that ? how this can be coded , I believe Self.StopLimitOrder does not do that , correct me if am wrong 

3- not related to the algo that I want to deploy , but still important for future research is how to actually find a faster way on QC to find say a certain set of stock tickers which happen to feature each in more than one ETF ?

By the way I have looked in to the forum but nothing came up as well as in the documentation

 
YOUR INPUT IS HIGHLY APPRECIATED 

Hi Ashraf

I cannot confirm since I have no guarantee that it must fit your usage, but I believe an R-16 research node would most likely be suitable for you. 3000 stocks with their options might get up to 6000 symbols, so possibly you'd still need to divide and conquer the task by 1000 stocks per instance for say.

As per the live node, it would depend on your algorithm's logic. Will the computational task be heavy? Do you run a speed-important algorithm? Will you do a lot of processing in a tight window? Do you handle a large amount of data (not symbols but data, including any in-algo processed ones with possible large size)? These are some thoughts on deciding which node suits your need. If the tasks and logic are simple and straightforward, I'd say an L-micro is enough though.

• On.schedule method to define time rules for entry and exit in a separate way 

You make 2 methods and call them with 2 scheduled events

• Order expiration time issue

Change the TimeInForce order property.

self.DefaultOrderProperties.TimeInForce = TimeInForce.GoodTilDate(self.Time + timedelta(minutes=5))
self.MarketOrder("SPY", 100)

• Fast filtering of constituents in more than 1 ETF

I'd say it is impossible for now. You'll need to loop through all ETFs to do so.

Best
Louis

HI Louis , 

does it mean R8-16 with 0 nodes ? and in case one more node added , will that allow access to more compute (including more symbols and contracts to cover as well as actual processing using the built-in function and methods within the platform while developing strategies) ? 

best 

Hi Ashraf

I was meaning 1 node of R8-16. Adding an additional node cannot make them work parallel in the same session. 2-node is meant to be 2 different research sessions working simultaneously.

Best
Louis