QUANTCONNECT COMMUNITY
I notice that sometimes my backtest runs very quickly and the server statistics show >100% CPU usage, but other times it runs extremely slowly with <10% CPU usage. This is for the exact same algorithm. Is it because I am using a Community B-MICRO instance? Or is it being bottlenecked when trying to load data? Will subscribing to QC and getting a beefier backtesting node help with this?
Thanks,
Karthik
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
QuantConnect™ 2024. All Rights Reserved
|
|
Share New Research 
Jared Broad
Hi Karthik. Correct the free tier is a shared-CPU, the speed achieved will depend entirely on other free user loads. If you subscribe to a paid node it's a dedicated CPU.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Karthik Kailash
Perfect, that is exactly the information I was looking for. Thank you!
Karthik Kailash
I am re-running my backtest on a dedicated node but it is still running very slowly with low CPU utilization. Is there something else that could be the bottleneck? Is the first run generally slower while the data populates? My backtest is running on SPY with minute resolution, and after 15 minutes it has only covered 4 months of simulation time.
Also, are free tier research nodes also shared CPU?
Jared Broad
Yes, free tier research nodes are also shared CPU's. Please post some code to review; it might be a bug in the algorithm causing it to slow down but it's hard to suggest solutions without knowing what you're trying to do.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Karthik Kailash
Jared Broad thanks for the clarification. I've pasted my code below. Now that I am on a dedicated node, it occasionally runs slow but now if I kill the backtest and start a new one the new one will usually runs at the expected fast level. Whereas on the free node the new test would usually be just as slow.
from Execution.ImmediateExecutionModel import ImmediateExecutionModel import pandas as pd class MyOptionMarginModel: option = None originalMarginModel = None def __init__(self, option, originalMarginModel): self.option = option self.originalMarginModel = originalMarginModel def GetLeverage(self, security): return self.originalMarginModel.GetLeverage(security) def SetLeverage(self, security, leverage): self.originalMarginModel.SetLeverage(security, leverage) def HasSufficientBuyingPowerForOrder(self, parameters): return self.originalMarginModel.HasSufficientBuyingPowerForOrder(parameters) def GetMaximumOrderQuantityForTargetBuyingPower(self, parameters): return self.originalMarginModel.GetMaximumOrderQuantityForTargetBuyingPower(parameters) def GetMaximumOrderQuantityForDeltaBuyingPower(self, parameters): return self.originalMarginModel.GetMaximumOrderQuantityForDeltaBuyingPower(parameters) def GetReservedBuyingPowerForPosition(self, params): if self.option.IsDelisted: return ReservedBuyingPowerForPosition(0) else: return self.originalMarginModel.GetReservedBuyingPowerForPosition(params) def GetBuyingPower(self, parameters): return self.originalMarginModel.GetBuyingPower(parameters) class CalibratedUncoupledRadiator(QCAlgorithm): ''' If SPY is within a certain % of its 52W high, write a call option that is a certain $ above the opening price. If the option gets assigned, cover ASAP ''' # Paramaters callOtmAmount = 6 withinYearlyHighPct = 10 spy = None spyOption = None spyCoverOrderTicket = None yearlyHigh = None days = 0 def Initialize(self): self.SetStartDate(2007, 1, 1) self.SetEndDate(2020, 11, 19) self.SetCash(100000) #self.callOtmAmount = int(self.GetParameter("callOtmAmount")) # TODO deal with missing parameter self.spy = self.AddEquity("SPY", Resolution.Minute) self.spy.SetDataNormalizationMode(DataNormalizationMode.Raw) # Required for working with options self.spy.SetLeverage(4) self.UniverseSettings.MinimumTimeInUniverse = 10 self.yearlyHigh = self.MAX(self.spy.Symbol, 252, Resolution.Daily) history = self.History(self.spy.Symbol, 252, Resolution.Daily) for bar in history.itertuples(): self.yearlyHigh.Update(bar.Index[1], bar.high) self.Debug("SPY yearly high warmed up, value = " + str(self.yearlyHigh.Current.Value)) self.Schedule.On(self.DateRules.Every(DayOfWeek.Monday, DayOfWeek.Wednesday, DayOfWeek.Friday), self.TimeRules.AfterMarketOpen("SPY", 1), self.OpeningBar) self.Schedule.On(self.DateRules.Every(DayOfWeek.Monday, DayOfWeek.Wednesday, DayOfWeek.Friday), self.TimeRules.BeforeMarketClose("SPY", 0), self.ClosingBar) self.SetExecution(ImmediateExecutionModel()) def OpeningBar(self): self.spyOption = None if not self.IsMarketOpen(self.spy.Symbol): self.Debug("Market not open, returning") return self.LogPortfolio() if self.Portfolio[self.spy.Symbol].Invested: self.Liquidate(self.spy.Symbol) ''' if self.spyCoverOrderTicket is None: self.Debug("Assigned, placing limit order to cover") self.spyCoverOrderTicket = self.LimitOrder(self.spy.Symbol, -1 * self.Portfolio["SPY"].Quantity, self.Portfolio["SPY"].AveragePrice) return ''' high = self.yearlyHigh.Current.Value if 100 * (high - self.Securities[self.spy.Symbol].Close)/high > self.withinYearlyHighPct: self.Debug("Too far away from yearly high: " + str(self.Securities[self.spy.Symbol].Close) + " vs " + str(high)) return options = self.OptionChainProvider.GetOptionContractList(self.spy.Symbol, self.Time) if options is None or len(options) == 0: self.Error("No options found") return callsExpiringToday = [o for o in options if o.ID.Date.date() == self.Time.date() and o.ID.OptionRight == OptionRight.Call] sortedByStrike = sorted(callsExpiringToday, key = lambda o: o.ID.StrikePrice) for option in sortedByStrike: id = option.ID if id.StrikePrice >= self.Securities[self.spy.Symbol].Open + self.callOtmAmount: self.spyOption = self.AddOptionContract(option, Resolution.Minute) mm = MyOptionMarginModel(self.spyOption, self.spyOption.MarginModel) self.spyOption.SetMarginModel(mm) self.Debug(str(id.Date) + " " + str(id.StrikePrice) + " " + str(id.OptionRight)) break ''' if self.days > 30: self.Quit() self.days = self.days+1 ''' def ClosingBar(self): self.spyOption = None def OnData(self, slice): for kvp in slice.Delistings: symbol = kvp.Key value = kvp.Value if value.Type == DelistingType.Warning: self.Debug("OnData(Delistings): {0}: {1} will be delisted at end of day today.".format(self.Time, symbol)) if value.Type == DelistingType.Delisted: self.Debug("OnData(Delistings): {0}: {1} has been delisted.".format(self.Time, symbol)) if self.spyOption is not None and self.Securities[self.spyOption.Symbol].Price != 0: self.MarketOrder(self.spyOption.Symbol, -5) self.spyOption = None return def OnOrderEvent(self, orderEvent): self.Debug(str(orderEvent)) oe = orderEvent #self.Debug("OrderID: " + str(oe.OrderId) + " EventID: " + str(oe.Id) + " Symbol: " + str(oe.Symbol) + " Status: " + str(oe.Status) + " FillPrice: " + str(oe.FillPrice) + " FillQuantity: " + str(oe.FillQuantity) + " IsAssignment: " + str(oe.IsAssignment) + " Message: " + oe.Message) if not self.spyCoverOrderTicket is None and orderEvent.OrderId == self.spyCoverOrderTicket.OrderId and self.spyCoverOrderTicket.Status == OrderStatus.Filled: self.Debug("covered!") self.spyCoverOrderTicket = None def LogPortfolio(self): if len(self.Portfolio.Keys) > 0: self.Debug("Portfolio TotalMarginUsed=" + str(self.Portfolio.TotalMarginUsed)) for symbol in self.Portfolio.Keys: holding = self.Portfolio[symbol] if holding.Invested: self.Debug(str(symbol) + " " + str(holding.Quantity) + " @ " + str(holding.AveragePrice) + " Delisted=" + str(self.Securities[holding.Symbol].IsDelisted))Pierre Vidal
Hi Karthik, Jared,
I often experienced the same problem recently, and today it seems systematic. Grateful for Karthik raising it.
Exactly as described above: " sometimes my backtest runs very quickly and the server statistics show >100% CPU usage, but other times it runs extremely slowly with <10% CPU usage ".
I experienced the issue randomly (apparently), with different algos. Most of the time, stopping / launching the backtest again (and/or sign out / sign in in between) was sufficient to get a normally running backtest. But not today.
I am not on a premium subscription (yet) and understand the consequences, but still I would not expect such a dramatic drop in available CPU - which really makes backtesting unbearably slow?
Thanks in advance for any guidance!
Jared Broad
Pierre Vidal you're on the free tier - this is a shared CPU. The speed depends on the number of clients using the free tier. If you'd like a predictable dedicated node, which is also 40% faster in clock speed, I highly recommend upgrading =). This also supports the platform to keep us sustainable.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Karthik Kailash
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!