System.OutOfMemoryException

Loading [MathJax]/jax/output/HTML-CSS/config.js

QuantConnect Community Discussions

QUANTCONNECT COMMUNITY

Join Our Discord Channel

Join QuantConnect's Discord server for real-time support, where a vibrant community of traders and developers awaits to help you with any of your QuantConnect needs.

Draft Discussions

Bookmarked Discussions

System.OutOfMemoryException

Last comment by - October 2024

System.OutOfMemoryException

"System.OutOfMemoryException: Your algorithm exceeded memory limits on QuantConnect."

Share New Research

Start New Discussion Sign up

SEARCH DISCUSSIONS

360,000 Quants.

Become a Quant

VOTE FOR UPCOMING FEATURES

Share your input and vote on our future direction.

LEAN Roadmap

JOIN OUR Community MAILING LIST

Create an account on QuantConnect for the latest community delivered to your inbox.

System.OutOfMemoryException

Dharmesh Khalasi | September 2024

Runtime Error: Exception of type 'System.OutOfMemoryException' was thrown. in TimeSliceFactory.cs:line 370

 def CoarseSelectionFunction(self, coarse: List[CoarseFundamental]) -> List[Symbol]:
        self.logger.log("CoarseSelectionFunction Started.")
        selected: List[Symbol] = []
        for stock in coarse:
            if stock.Price > parameters.PRICE_THRESHOLD and stock.DollarVolume > parameters.DOLLAR_VOLUME_THRESHOLD:
                selected.append(stock.Symbol)
                # Initialize the historical data if not already done
                # if stock.Symbol not in self.history:
                #     self.InitializeIndicators(stock.Symbol)
        self.logger.log(f"CoarseSelectionFunction Selected {len(selected)} stocks.")      
        return selected

Author

Dharmesh Khalasi

September 2024

Upvote

Mia Alissi INVESTOR

September 2024

To address the System.OutOfMemoryException and improve the CoarseSelectionFunction, you need to make the following changes:

Correct Attribute Names: The attributes Price, DollarVolume, and Symbol should be lowercase. Use stock.price, stock.dollar_volume, and stock.symbol.
Define Parameters: Ensure that parameters.PRICE_THRESHOLD and parameters.DOLLAR_VOLUME_THRESHOLD are defined and accessible within the function. You might need to define these parameters in your class or pass them as arguments.

Here's the corrected version of your function:

def CoarseSelectionFunction(self, coarse: List[CoarseFundamental]) -> List[Symbol]:
    self.logger.log("CoarseSelectionFunction Started.")
    selected: List[Symbol] = []
    for stock in coarse:
        if stock.price > self.price_threshold and stock.dollar_volume > self.dollar_volume_threshold:
            selected.append(stock.symbol)
            # Initialize the historical data if not already done
            # if stock.symbol not in self.history:
            #     self.InitializeIndicators(stock.symbol)
    self.logger.log(f"CoarseSelectionFunction Selected {len(selected)} stocks.")      
    return selected

Additional Recommendations:

Optimize Memory Usage: Consider filtering the stocks further before adding them to the selected list to reduce memory usage.
Batch Processing: If possible, process stocks in smaller batches to avoid loading too much data into memory at once.
Monitor Memory Usage: Use logging to track memory usage and identify any operations that consume excessive memory.

Sources: Legacy Fundamental Universes - QuantConnect.com, Stock Selection Strategy Based On Fundamental Factors, Fundamental Factor Long Short Strategy, Momentum Short Term Reversal Strategy, Coarse Fine Fundamental Combo Algorithm

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Upvote

Dharmesh Khalasi INVESTOR

September 2024

How to do batch process to avoid to load too many data into memory. write sample code
How to track memory usage, write sample code

Upvote

1 person upvoted this

Louis Szeto INVESTOR

QuantConnect | October 2024

Hi Dharmesh

Do several rounds of backtest with splitted selections (like each algorithm selects the tickers start with “A”/"B"/…)
Your cloud algorithm will not be able to handle this, but you can take a look at the sidebar's performance.

It is recommended to upgrade your backtest node if you have a large universe to work with.

Best
Louis

Upvote

Alexandre Catarino INVESTOR

QuantConnect | August 2017

AddEquity creates a Security object and subcribes to data updates. In the process of creating a Security, a Symbol object is also created, it is added to a cache so that we can use either the Symbol object or the string (implicit conversion) in all the methods that required a Symbol object as input.
Please add a concrete example of the trouble you are having. Let's make it work!

Upvote

1 person upvoted this

Pranava INVESTOR

April 2018

How about python? Is there a equivalent for "userdefineduniverse" for python? Or is there a "finefiltereduniverse forpython"?

Upvote

Michael Manus INVESTOR

April 2018

https://www.quantconnect.com/forum/discussion/3682/algo-framework-etf-momentum-rebalancing-using-mean-variance-optimizationhttps://github.com/QuantConnect/Lean/blob/56bbc4c3da747dcf021c8221eb54b43ad075b3d7/Algorithm.Python/ConstituentsQC500GeneratorAlgorithm.pyhttps://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/UniverseSelectionRegressionAlgorithm.pyhttps://github.com/QuantConnect/Lean/blob/cbd953437f987392f2c2e154adcb7d43d684b7f1/Algorithm.Python/CoarseFineFundamentalComboAlgorithm.pyhttps://github.com/QuantConnect/Lean/blob/cbd953437f987392f2c2e154adcb7d43d684b7f1/Algorithm.Python/EmaCrossUniverseSelectionAlgorithm.py

Upvote

Wu zheng INVESTOR

January 2019

I may have this figured out. When AddUniverse(CoarseSelectionFunction,FineSelectionFunction is called in Initialization(). the two filter functions are registered, but the universe is NOT created immediately. The universe management data structure is populated later execution exits Initialization(). As a result, you cannot access universe.Members.Keys in Initialization().

The following shows a work around. They are pieces of my code with editing. (I have less than a month experience with C# and Quantconnect, so excuse me if something does not look normal);

public override void Initialize()
{

//code removed

AddUniverse(CoarseSelectionFunction, FineSelectionFunction);

}

public void InitializationPart2()

        {

           // Use UniverseManager to find symbols from the Universe Selection
           foreach (var universe in UniverseManager.Values)
           {
                if (universe is UserDefinedUniverse)
                {
                    continue;
                }

                List<Symbol> symbols = new List<Symbol> (universe.Members.Keys);

                Debug(string.Join(", ",symbols.Select(x => x.Value.ToString())));

                foreach (Symbol symbol in symbols)
               {
                    //code removed
                   Debug(symbol.Value);
                    /code removed
               }

           }

}

        bool firstTimeCallOnData=true;
        public void OnData(TradeBars data)
        {

           if (firstTimeCallOnData) InitializationPart2();
           firstTimeCallOnData=false;

}

Upvote

1 person upvoted this

Wu zheng INVESTOR

January 2019

The above work around does work

2016-01-05 00:00:00 find universe: QuantConnect.Data.UniverseSelection.FineFundamentalFilteredUniverse 2016-01-05 00:00:00 CPB, DAL, AAL, SJM, GIS, ALK, LUV, K, UAL, JBLU, SAVE, KHC 2

Upvote

1 person upvoted this

Jared Broad INVESTOR

QuantConnect | January 2019

100% correct wu zheng! Nice work!

Upvote

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

360,000 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

Additional Recommendations:

IN THIS RESEARCH

PARTICIPANTS

Actions

Join QuantConnect for Free

Platform

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

360,000 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

Additional Recommendations:

IN THIS RESEARCH

PARTICIPANTS

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

SHARE

Actions

Join QuantConnect for Free