Share New Research 
Hey there,
I'm trying to fetch indicator values on a cloud research notebook for about 3000+ different combinations of symbol and time (total of 90 stocks), the indicator is RSI for example with 100 periods.
The thing is that after a few minutes an OutOfMemoryException shows up, running on an R-MICRO node (2GB RAM). I tried upgrading the research node to R1-4 (4GB RAM) but got the same problem.
Few questions if I may:
- Is there a way to view a research node's realtime resource consumption (CPU, memory) ?
- Is there a list of limitations of the research nodes somewhere?
- I noticed the “Datasets Subscriptions” in the pricing page, what exactly does it allows? would I be able to download a 1-minute data for many stocks and research locally?
Any advise on how to workaround this limitation will be appreciated, partitioning is considered. Below is the full exception.
Thank you!
OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown. at System.Collections.Generic.List`1.set_Capacity(Int32 value) at System.Collections.Generic.List`1.AddWithResize(T item) at QuantConnect.Research.QuantBook.<>c__DisplayClass25_0.<Indicator>b__3(Object s, IndicatorDataPoint e) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 722 at QuantConnect.Indicators.IndicatorBase`1.Update(IBaseData input) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Indicators/IndicatorBase.cs:line 93 at QuantConnect.Indicators.IndicatorBase`1.Update(DateTime time, Decimal value) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Indicators/IndicatorBase.cs:line 134 at QuantConnect.Research.QuantBook.<>c__DisplayClass25_0.<Indicator>b__5(BaseData bar) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 731 at QuantConnect.Data.SliceExtensions.PushThrough(IEnumerable`1 slices, Action`1 handler) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Common/Data/SliceExtensions.cs:line 222 at QuantConnect.Research.QuantBook.Indicator(IndicatorBase`1 indicator, IEnumerable`1 history, Func`2 selector) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 728 at QuantConnect.Research.QuantBook.Indicator(IndicatorBase`1 indicator, Symbol symbol, DateTime start, DateTime end, Nullable`1 resolution, Func`2 selector) in /LeanCloud/CI.Builder/bin/Debug/src/QuantConnect/Lean/Research/QuantBook.cs:line 549
Varad Kabade
Hi Erez Tison,
OutOfMemoryException occurs when you have more data in the memory than is available (4 gb). This can be avoided by optimizing the code; please share the attached backtest for more assistance
Is there a way to view a research node's realtime resource consumption (CPU, memory) ?
No currently, we cannot view the memory usage
Is there a list of limitations of the research nodes somewhere?
In the pricing page, you can find the research node limits for Free,Quant Researcher,Team,Trading Firm and Institution plan
I noticed the “Datasets Subscriptions” in the pricing page, what exactly does it allows? would I be able to download a 1-minute data for many stocks and research locally?
Yes, with a data subscription, you can use the data for research and develop algorithms locally at all the available resolutions. You can add the Security Master dataset to have all the US Equity data, or can download individual files from the Data Explorer with the CLI.Best,
Varad Kabade
Erez Tison
Thank you Varad kabade for the prompt answer.
Assuming the research node memory limit was doubled when I upgraded to from R-MICRO node (2GB RAM) to R1-4 (4GB RAM), and the problem persist, is there another per-kernel limitation or something similar that is not enlarged in an upgrade?
The code is really simple, just qb.indicator for RSI with 110 periods, I did noticed that lesser periods may avoid this problem.
Thank you.
Varad Kabade
Hi Erez,
We recommend trying to perform your analysis/research in batch (some stocks at a time) and storing the result in ObjectStore, and reviewing them as needed. This would make sure that we don't exhaust our working memory. Would you please share a dedicated backtest containing the research notebook for further assistance? Refer to the following thread regarding storing and retrieving data to/from the ObjectStore.
Best,
Varad Kabade
Erez Tison
Thank you Varad, will try partitioning and ObjectStore.
Best, Erez.
Erez Tison
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!