Custom data. Out of memory exception

Hello,

I would like to use custom data such as $SECReport10K$ in my algorithms, but I get an out of memory exception as soon as I use more than a few handful of securities.

I add the data with $AddData$ .

Is there anything I should do to free memory when the security is removed from the universe, or is custom data unsubscription taken care of automatically?

Has anybody had better luck?
Looking forward to your thoughts.

public override void Initialize()
{
    SetStartDate(2018, 1, 12);  //Set Start Date
    SetCash(100000);             //Set Strategy Cash
    AddUniverse(Universe.DollarVolume.Top(50)); // creashes already with 30 or less
}
public override void OnData(Slice data)
{
    foreach (BaseData baseData in data.Values.Where(sym => sym.Symbol.IsCustomDataType<SECReport10K>())) {
        Debug($"{Time}: {baseData}");
    }
}
public override void OnSecuritiesChanged(SecurityChanges changes)
{
    base.OnSecuritiesChanged(changes);
    foreach (Security security in changes.AddedSecurities) {
        AddData<SECReport10K>(security.Symbol);
    }
    // foreach (Security security in changes.RemovedSecurities) {
    //     .. is there something we should do to remove custom data?
    // }
}
+ Expand
- Collapse

public override void OnSecuritiesChanged(SecurityChanges changes)
{
foreach (var symbol in changes.AddedSecurities.Select(x => x.Symbol))
{
// If added to universe, adds SECReport8K
_mappings[symbol] = AddData<SECReport8K>(symbol).Symbol;
}
foreach (var symbol in changes.RemovedSecurities.Select(x => x.Symbol))
{
// If removed from the universe, liquidate and remove the custom data from the algorithm
Liquidate(symbol, "Removed from Universe");
Symbol reportSymbol;
if (_mappings.TryGetValue(symbol, out reportSymbol))
{
RemoveSecurity(reportSymbol);
_mappings.Remove(symbol);
}
}
}

+ Expand

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Alexandre Catarino INVESTOR

QuantConnect | July 2022

Hi R G,

When we add alternative data that is linked to a Security, we can save the alternative data Symbol in a dictionary keyed by the underlying Symbol and use that alternative data Symbol to remove its subscription when the underlying is removed:

        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            foreach (var symbol in changes.AddedSecurities.Select(x => x.Symbol))
            {
                // If added to universe, adds SECReport8K
                _mappings[symbol] = AddData<SECReport8K>(symbol).Symbol;
            }
            foreach (var symbol in changes.RemovedSecurities.Select(x => x.Symbol))
            {
                // If removed from the universe, liquidate and remove the custom data from the algorithm
                Liquidate(symbol, "Removed from Universe");
                
                Symbol reportSymbol;
                if (_mappings.TryGetValue(symbol, out reportSymbol))
                {
                    RemoveSecurity(reportSymbol);
                    _mappings.Remove(symbol);
                }
            }
        } 
+ Expand
- Collapse

Please find the fully working example on the US SEC Filings dataset page.

Best regards,
Alex

Upvote

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

368,200 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

Actions

Join QuantConnect for Free