Question regarding Coarse Universe Selection and lean CLI

Loading web-font TeX/Math/Italic

QuantConnect Community Discussions

QUANTCONNECT COMMUNITY

Join Our Discord Channel

Join QuantConnect's Discord server for real-time support, where a vibrant community of traders and developers awaits to help you with any of your QuantConnect needs.

Draft Discussions

Bookmarked Discussions

Question regarding Coarse Universe Selection and lean CLI

Last comment by - March 2023

Question regarding Coarse Universe Selection and lean CLI

User has a question about Coarse Universe Selection and lean CLI, specifically regarding minute data access.

Share New Research

Start New Discussion Sign up

SEARCH DISCUSSIONS

367,400 Quants.

Become a Quant

VOTE FOR UPCOMING FEATURES

Share your input and vote on our future direction.

LEAN Roadmap

JOIN OUR Community MAILING LIST

Create an account on QuantConnect for the latest community delivered to your inbox.

Question regarding Coarse Universe Selection and lean CLI

Super Richman | January 2023

Thanks for adding $failed-data-requests$ and $succeeded-data-requests$ files to lean cli backtests, much helpful!

I am building a strat that using $CoarseSelectionFunction$ and executing in minute data.

I have noticed in the $failed-data-requests$ file that whenever a ticker is once added to the Universe then LEAN is trying to access its minute data even though it is not actually in the current trading day (i.e. removed for today).
I am downloading only the necessary minute data selected by the previous day DollarVolume so I don't have all of the tickers minute data.

Why is that happening?

Author

Super Richman

January 2023

Upvote

Louis Szeto INVESTOR

QuantConnect | January 2023

Hi Super Richman

You'll also need the coarse universe data downloaded for that functionality. Either use the cli data downloader helper to download in advance, or download the needed data alongside with the backtest.

Best
Louis

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

Upvote

Super Richman INVESTOR

January 2023

Thanks Louis. Coarse folder is in place as well as custom mapping and factor files according to the tickmap specification. the algo purpose is just to test which data is accessed since I am only downloading the relevant minute data.

/*
 * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
 * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
*/
using QuantConnect.Interfaces;
using System.Collections.Generic;
using System.Linq;
using QuantConnect.Data.Market;
using QuantConnect.Data.UniverseSelection;
using QuantConnect.Orders;
using System.IO;
using System.Collections;
using System;
using QuantConnect.Packets;
namespace QuantConnect.Algorithm.CSharp
{
    /// <summary>
    /// In this algorithm we demonstrate how to use the coarse fundamental data to
    /// define a universe as the top dollar volume
    /// </summary>
    /// <meta name="tag" content="using data" />
    /// <meta name="tag" content="universes" />
    /// <meta name="tag" content="coarse universes" />
    /// <meta name="tag" content="regression test" />
    public class CoarseFundamentalTop3Algorithm : QCAlgorithm, IRegressionAlgorithmDefinition
    {
        private const int NumberOfSymbols = 9;       
        private const string strFolderName = "QCiSAweSome";
        // initialize our changes to nothing
        private SecurityChanges _changes = SecurityChanges.None;
        TradingCalendar tc;
        ArrayList alCurrTickers;
        public override void Initialize()
        {
            alCurrTickers = new ArrayList();
            UniverseSettings.Resolution = Resolution.Minute;
            SetStartDate(2023, 1, 4); // Set Start Date // Day before since our universe need to be saved
            SetEndDate(2023, 1, 14); // Set end Date
            SetCash(100000);
            // this add universe method accepts a single parameter that is a function that
            // accepts an IEnumerable<CoarseFundamental> and returns IEnumerable<Symbol>
            AddUniverse(CoarseSelectionFunction);
        }
        // sort the data by daily dollar volume and take the top 'NumberOfSymbols'
        public static IEnumerable<Symbol> CoarseSelectionFunction(IEnumerable<CoarseFundamental> coarse)
        {
            // sort descending by daily dollar volume
            var sortedByDollarVolume = coarse.OrderByDescending(x => x.DollarVolume);
            // take the top entries from our sorted collection
            var top = sortedByDollarVolume.Take(NumberOfSymbols);
            var symbols = top.Select(x => x.Symbol.Value);            
            
            // we need to return only the symbol objects
            return top.Select(x => x.Symbol);
        }
        //Data Event Handler: New data arrives here. "TradeBars" type is a dictionary of strings so you can access it by symbol.
        public void OnData(TradeBars data)
        {
            // This algo is just for data check so nothing to do here
        }
        // this event fires whenever we have changes to our universe
        public override void OnSecuritiesChanged(SecurityChanges changes)
        {
            Console.WriteLine("Added:");
            foreach (var added in changes.AddedSecurities) { Console.WriteLine(added.Symbol.ID.ToString() + "\t" + added.Symbol.Value.ToString()); }
            Console.WriteLine("Removed:");
            foreach (var removed in changes.RemovedSecurities) { Console.WriteLine(removed.Symbol.ID.ToString() + "\t" + removed.Symbol.Value.ToString()); }
        }
        public override void OnOrderEvent(OrderEvent fill)
        {
           // Log($"OnOrderEvent({UtcTime:o}):: {fill}");
        }
        /// <summary>
        /// This is used by the regression test system to indicate if the open source Lean repository has the required data to run this algorithm.
        /// </summary>
        public bool CanRunLocally { get; } = true;
        /// <summary>
        /// This is used by the regression test system to indicate which languages this algorithm is written in.
        /// </summary>
        public Language[] Languages { get; } = { Language.CSharp, Language.Python };
        /// <summary>
        /// Data Points count of all timeslices of algorithm
        /// </summary>
        public long DataPoints => 78091;
        /// <summary>
        /// Data Points count of the algorithm history
        /// </summary>
        public int AlgorithmHistoryDataPoints => 0;
        /// <summary>
        /// This is used by the regression test system to indicate what the expected statistics are from running the algorithm
        /// </summary>
        public Dictionary<string, string> ExpectedStatistics => new Dictionary<string, string>
        {
            {"Total Trades", "12"},
            {"Average Win", "0.55%"},
            {"Average Loss", "-0.26%"},
            {"Compounding Annual Return", "16.717%"},
            {"Drawdown", "1.700%"},
            {"Expectancy", "0.850"},
            {"Net Profit", "0.637%"},
            {"Sharpe Ratio", "1.088"},
            {"Probabilistic Sharpe Ratio", "50.223%"},
            {"Loss Rate", "40%"},
            {"Win Rate", "60%"},
            {"Profit-Loss Ratio", "2.08"},
            {"Alpha", "0.198"},
            {"Beta", "0.741"},
            {"Annual Standard Deviation", "0.118"},
            {"Annual Variance", "0.014"},
            {"Information Ratio", "2.294"},
            {"Tracking Error", "0.097"},
            {"Treynor Ratio", "0.173"},
            {"Total Fees", "$27.94"},
            {"Estimated Strategy Capacity", "$200000000.00"},
            {"Lowest Capacity Asset", "AAPL R735QTJ8XC9X"},
            {"Fitness Score", "0.28"},
            {"Kelly Criterion Estimate", "0"},
            {"Kelly Criterion Probability Value", "0"},
            {"Sortino Ratio", "3"},
            {"Return Over Maximum Drawdown", "9.559"},
            {"Portfolio Turnover", "0.308"},
            {"Total Insights Generated", "0"},
            {"Total Insights Closed", "0"},
            {"Total Insights Analysis Completed", "0"},
            {"Long Insight Count", "0"},
            {"Short Insight Count", "0"},
            {"Long/Short Ratio", "100%"},
            {"Estimated Monthly Alpha Value", "$0"},
            {"Total Accumulated Estimated Alpha Value", "$0"},
            {"Mean Population Estimated Insight Value", "$0"},
            {"Mean Population Direction", "0%"},
            {"Mean Population Magnitude", "0%"},
            {"Rolling Averaged Population Direction", "0%"},
            {"Rolling Averaged Population Magnitude", "0%"},
            {"OrderListHash", "de456413f89396bd6f920686219ed0a5"}
        };
    }
}
+ Expand
- Collapse

Upvote

Louis Szeto INVESTOR

QuantConnect | February 2023

Hi Superrich Man

Sorry for the delay. Could you please also tell us which files you have in your local data directory? It will help a lot on the debugging since we cannot reproduce your problem.

Best
Louis

Upvote

Super Richman INVESTOR

February 2023

Hi Louis,

Thanks for replying! Delay is always good, I hope you were on vacation or did something fun ;)

I am adding some files that will help you understand, I hope it will be good for you.

succeeded-data-requests-20230118140038721.txt

/equity/usa/fundamental/coarse/20230103.csv
/equity/usa/fundamental/coarse/20230104.csv
/equity/usa/fundamental/coarse/20230105.csv
/equity/usa/fundamental/coarse/20230106.csv
/equity/usa/fundamental/coarse/20230109.csv
/equity/usa/fundamental/coarse/20230110.csv
/equity/usa/fundamental/coarse/20230111.csv
/equity/usa/fundamental/coarse/20230112.csv
/equity/usa/fundamental/coarse/20230113.csv
/equity/usa/minute/aapl/20230104_trade.zip
/equity/usa/minute/pg/20230104_trade.zip
/equity/usa/minute/pg/20230104_quote.zip
/equity/usa/minute/aapl/20230104_quote.zip
/equity/usa/minute/msft/20230104_quote.zip
/equity/usa/minute/tsla/20230104_trade.zip
/equity/usa/minute/cvx/20230104_quote.zip
/equity/usa/minute/amzn/20230104_trade.zip
/equity/usa/minute/msft/20230104_trade.zip
/equity/usa/minute/tsla/20230104_quote.zip
/equity/usa/minute/amzn/20230104_quote.zip
/equity/usa/minute/meta/20230104_quote.zip
/equity/usa/minute/meta/20230104_trade.zip
/equity/usa/minute/nvda/20230104_quote.zip
/equity/usa/minute/cvx/20230104_trade.zip
/equity/usa/minute/nvda/20230104_trade.zip
/equity/usa/minute/meta/20230105_trade.zip
/equity/usa/minute/nvda/20230105_quote.zip
/equity/usa/minute/amzn/20230105_trade.zip
/equity/usa/minute/cvx/20230105_trade.zip
/equity/usa/minute/nvda/20230105_trade.zip
/equity/usa/minute/meta/20230105_quote.zip
/equity/usa/minute/tsla/20230105_trade.zip
/equity/usa/minute/cvx/20230105_quote.zip
/equity/usa/minute/msft/20230105_quote.zip
/equity/usa/minute/tsla/20230105_quote.zip
/equity/usa/minute/amzn/20230105_quote.zip
/equity/usa/minute/aapl/20230105_quote.zip
/equity/usa/minute/pg/20230109_quote.zip
/equity/usa/minute/pg/20230109_trade.zip
/equity/usa/minute/jnj/20230104_trade.zip
/equity/usa/minute/jnj/20230104_quote.zip
/equity/usa/minute/nvda/20230106_trade.zip
/equity/usa/minute/amzn/20230106_trade.zip
/equity/usa/minute/meta/20230106_quote.zip
/equity/usa/minute/tsla/20230106_quote.zip
/equity/usa/minute/msft/20230106_quote.zip
/equity/usa/minute/nvda/20230106_quote.zip
/equity/usa/minute/cvx/20230106_trade.zip
/equity/usa/minute/tsla/20230106_trade.zip
/equity/usa/minute/msft/20230105_trade.zip
/equity/usa/minute/amzn/20230106_quote.zip
/equity/usa/minute/cvx/20230106_quote.zip
/equity/usa/minute/aapl/20230105_trade.zip
/equity/usa/minute/aapl/20230106_quote.zip
/equity/usa/minute/meta/20230106_trade.zip
/equity/usa/hour/spy.zip
/equity/usa/minute/jnj/20230105_trade.zip
/equity/usa/minute/jnj/20230105_quote.zip
/equity/usa/minute/nvda/20230109_trade.zip
/equity/usa/minute/amzn/20230109_trade.zip
/equity/usa/minute/tsla/20230109_quote.zip
/equity/usa/minute/msft/20230109_quote.zip
/equity/usa/minute/cvx/20230109_trade.zip
/equity/usa/minute/nvda/20230109_quote.zip
/equity/usa/minute/tsla/20230109_trade.zip
/equity/usa/minute/meta/20230109_quote.zip
/equity/usa/minute/cvx/20230109_quote.zip
/equity/usa/minute/amzn/20230109_quote.zip
/equity/usa/minute/meta/20230109_trade.zip
/equity/usa/minute/aapl/20230109_quote.zip
/equity/usa/minute/aapl/20230106_trade.zip
/equity/usa/minute/msft/20230106_trade.zip
/equity/usa/minute/jnj/20230106_quote.zip
/equity/usa/minute/jnj/20230106_trade.zip
/equity/usa/minute/v/20230105_trade.zip
/equity/usa/minute/v/20230105_quote.zip
/equity/usa/minute/v/20230106_trade.zip
/equity/usa/minute/v/20230106_quote.zip
/equity/usa/minute/msft/20230109_trade.zip
/equity/usa/minute/tsla/20230110_trade.zip
/equity/usa/minute/amzn/20230110_quote.zip
/equity/usa/minute/meta/20230110_trade.zip
/equity/usa/minute/cvx/20230110_quote.zip
/equity/usa/minute/aapl/20230110_quote.zip
/equity/usa/minute/nvda/20230110_trade.zip
/equity/usa/minute/jnj/20230110_trade.zip
/equity/usa/minute/jnj/20230110_quote.zip
/equity/usa/minute/amzn/20230110_trade.zip
/equity/usa/minute/tsla/20230110_quote.zip
/equity/usa/minute/msft/20230110_quote.zip
/equity/usa/minute/cvx/20230110_trade.zip
/equity/usa/minute/v/20230109_trade.zip
/equity/usa/minute/aapl/20230109_trade.zip
/equity/usa/minute/v/20230109_quote.zip
/equity/usa/minute/aapl/20230111_quote.zip
/equity/usa/minute/nvda/20230111_trade.zip
/equity/usa/minute/aapl/20230110_trade.zip
/equity/usa/minute/msft/20230111_quote.zip
/equity/usa/minute/cvx/20230111_trade.zip
/equity/usa/minute/amzn/20230111_trade.zip
/equity/usa/minute/meta/20230110_quote.zip
/equity/usa/minute/tsla/20230111_quote.zip
/equity/usa/minute/nvda/20230110_quote.zip
/equity/usa/minute/msft/20230110_trade.zip
/equity/usa/minute/v/20230110_quote.zip
/equity/usa/minute/v/20230110_trade.zip
/equity/usa/minute/aapl/20230111_trade.zip
/equity/usa/minute/msft/20230111_trade.zip
/equity/usa/minute/pg/20230111_trade.zip
/equity/usa/minute/pg/20230111_quote.zip
/equity/usa/minute/cvx/20230111_quote.zip
/equity/usa/minute/meta/20230111_trade.zip
/equity/usa/minute/aapl/20230112_quote.zip
/equity/usa/minute/nvda/20230112_trade.zip
/equity/usa/minute/aapl/20230112_trade.zip
/equity/usa/minute/amzn/20230112_trade.zip
/equity/usa/minute/cvx/20230112_trade.zip
/equity/usa/minute/tsla/20230112_quote.zip
/equity/usa/minute/msft/20230112_quote.zip
/equity/usa/minute/msft/20230112_trade.zip
/equity/usa/minute/meta/20230112_trade.zip
/equity/usa/minute/cvx/20230112_quote.zip
/equity/usa/minute/tsla/20230111_trade.zip
/equity/usa/minute/nvda/20230111_quote.zip
/equity/usa/minute/amzn/20230111_quote.zip
/equity/usa/minute/v/20230111_trade.zip
/equity/usa/minute/amzn/20230112_quote.zip
/equity/usa/minute/v/20230111_quote.zip
/equity/usa/minute/meta/20230111_quote.zip
/equity/usa/minute/tsla/20230112_trade.zip
/equity/usa/minute/meta/20230112_quote.zip
/equity/usa/minute/cvx/20230113_quote.zip
/equity/usa/minute/nvda/20230112_quote.zip
/equity/usa/minute/v/20230112_trade.zip
/equity/usa/minute/amzn/20230113_quote.zip
/equity/usa/minute/aapl/20230113_trade.zip
/equity/usa/minute/nvda/20230113_trade.zip
/equity/usa/minute/v/20230112_quote.zip
/equity/usa/minute/amzn/20230113_trade.zip
/equity/usa/minute/cvx/20230113_trade.zip
/equity/usa/minute/pg/20230112_trade.zip
/equity/usa/minute/msft/20230113_trade.zip
/equity/usa/minute/tsla/20230113_quote.zip
/equity/usa/minute/meta/20230113_trade.zip
/equity/usa/minute/pg/20230112_quote.zip
/equity/usa/minute/msft/20230113_quote.zip
/equity/usa/minute/tsla/20230113_trade.zip
/equity/usa/minute/pg/20230113_trade.zip
/equity/usa/minute/nvda/20230113_quote.zip
/equity/usa/minute/meta/20230113_quote.zip
/equity/usa/minute/pg/20230113_quote.zip
/equity/usa/minute/aapl/20230113_quote.zip
/equity/usa/minute/aal/20230113_trade.zip
/equity/usa/minute/aal/20230113_quote.zip
+ Expand
- Collapse

failed-data-requests-20230118140038722.txt

/equity/usa/minute/pg/20230105_trade.zip
/equity/usa/minute/pg/20230105_quote.zip
/equity/usa/minute/pg/20230106_quote.zip
/equity/usa/minute/pg/20230106_trade.zip
/equity/usa/minute/jnj/20230109_trade.zip
/equity/usa/minute/jnj/20230109_quote.zip
/equity/usa/minute/pg/20230110_trade.zip
/equity/usa/minute/pg/20230110_quote.zip
/equity/usa/minute/jnj/20230111_trade.zip
/equity/usa/minute/jnj/20230111_quote.zip
/equity/usa/minute/jnj/20230112_trade.zip
/equity/usa/minute/jnj/20230112_quote.zip
/equity/usa/minute/jnj/20230113_trade.zip
/equity/usa/minute/jnj/20230113_quote.zip
/equity/usa/minute/v/20230113_trade.zip
/equity/usa/minute/v/20230113_quote.zip

data-monitor-report-20230118140045884.json

{"succeeded-data-requests-count":154,"failed-data-requests-count":16,"succeeded-universe-data-requests-count":9,"failed-universe-data-requests-count":0,"total-data-requests-count":170,"failed-data-requests-percentage":9.0,"total-universe-data-requests-count":9,"failed-universe-data-requests-percentage":0.0,"data-request-rates":[0.0]}

Partial Log:

2023-01-04 09:31:00 2023-01-04
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	AAPL SRW990ATUCMD	126.8900
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	AMZN SRW990ATUCMD	86.5100
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	CVX SRW990ATUCMD	170.9500
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	JNJ SRW990ATUCMD	178.8900
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	FB V6OIPNZEM8V9	127.3800
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	MSFT SRW990ATUCMD	232.0200
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	NVDA SRW990ATUCMD	145.6600
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	PG SRW990ATUCMD	151.2500
2023-01-04 09:31:00 1/4/2023 9:30:00 AM	TSLA UNV35CB9Z6G5	109.1100
2023-01-05 00:00:00 Added:
2023-01-05 00:00:00 V U12VRGLO8PR9	V
2023-01-05 00:00:00 Removed:
2023-01-05 00:00:00 PG SRW990ATUCMD	PG
2023-01-05 09:31:00 2023-01-05

As deducted from log, for 5.1 PG is not needed, and yet we have a failed data request for minute data for PG

I have all the files needed per day of Coarse by DollarVolume - so the coarse slected top 10 minute data will be downloaded for tomorrow.

All of the “needed” data is in place, the failed requests are for days that are not in the coarse and should not actually be requested.

To reproduce:

Create a strat which uses coarse selection
load a minute data the day after (automatically by coarse strat)
the day after the stock was “removed” see the failed (or succeeded) data request for minute data even if unnecessary.

Feel free to correct me if I am wrong :P What do you think?

Upvote

Louis Szeto INVESTOR

QuantConnect | February 2023

Hi Super Richman

Do you have the “failed requests” files in position? Maybe you'll also need $lean data download … --overwrite$ if some existing/random generated files are there.

Best
Louis

Upvote

Super Richman INVESTOR

February 2023

Hi again Louis, I deliberately do not have the unneeded files since they are not required for the strategy.

Upvote

Louis Szeto INVESTOR

QuantConnect | March 2023

Hi Super Richman

I mean you have the needed files, but are they holding the data required? Like $equity/usa/daily/spy.csv$ , it might only hold a period of data which is insufficient for your backtest. You'll need the --overwrite option to download and replace this outdated file.

Best
Louis

Upvote

Author:

January 2023

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

367,400 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

Actions

Join QuantConnect for Free

Platform

SIGN IN

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

367,400 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

SHARE RESEARCH

SHARE DISCUSSION

SHARE ARTICLE

SHARE

Actions

Join QuantConnect for Free