Import UCITS ETF Data from Yahoo Finance

I've written this class to parse UCITS ETF data from Yahoo Finance, but I get the following error trying to do a backtest, what's wrong?

class UcitsEtf(PythonData):
    def __init__(self, start_date):
        self.start_date = start_date
    def GetSource(self, config, date, isLiveMode):
        ticker = config.Symbol.Value  
        start = int(time.mktime(datetime.strptime(self.start_date, "%Y-%m-%d").timetuple())) 
        end = int(time.time()) 
        url = f"https://query1.finance.yahoo.com/v7/finance/download/{ticker}?period1={start}&period2={end}&interval=1d&events=history&includeAdjustedClose=true"
        return SubscriptionDataSource(url, SubscriptionTransportMedium.RemoteFile)
    def handle_nulls(self, df):
        df.replace("null", np.nan, inplace=True) 
        df.fillna(df.rolling(2, min_periods=1).mean(), inplace=True) 
        return df
    def Reader(self, config, line, date, isLiveMode):
        index = UcitsEtf(self.start_date)
        index.Symbol = config.Symbol
        try:
            data = line.split(',')
            if data[0] == "Date":  
                return None
            data = self.handle_nulls(pd.DataFrame(data[1:], columns=["open", "high", "low", "close", "adj_close", "volume"])) 
            index.Time = datetime.strptime(data[0], "%Y-%m-%d")
            index.EndTime = index.Time + timedelta(days=1)
            index.Value = decimal.Decimal(data["close"])  
            index["open"] = float(data["open"])
            index["high"] = float(data["high"])
            index["low"] = float(data["low"])
            index["close"] = float(data["close"])
            index["adj_close"] = float(data["adj_close"])
            index["volume"] = int(data["volume"])
        except Exception as e:          
            print(f"Error: {line}")
            print(f"Exception: {e}")
            return None
        return index
+ Expand
- Collapse

self.TEST = self.AddData(UcitsEtf("01-01-2022"), "XS2L.MI", Resolution.Daily).Symbol

Error Message
During the algorithm initialization, the following exception has occurred: Index was outside the bounds of the array.
  at Initialize
    self.TEST = self.AddData(UcitsEtf("01-01-2022") in main.py: line 84
 Index was outside the bounds of the array.

try:
data = line.split(',')
if data[0] == "Date":
return None
index.Time = datetime.strptime(data[0], "%Y-%m-%d")
index.EndTime = index.Time + timedelta(days=1)
index.Value = decimal.Decimal(data[4])
index["open"] = float(data[1])
index["high"] = float(data[2])
index["low"] = float(data[3])
index["close"] = float(data[4])
index["adj_close"] = float(data[5])
index["volume"] = int(data[6])
except Exception as e:
print(f"Error: {line}")
print(f"Exception: {e}")
return None

+ Expand

The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.

try:
data = line.split(',')
if data[0] == "Date":
return None
index.Time = datetime.strptime(data[0], "%Y-%m-%d")
index.EndTime = index.Time + timedelta(days=1)
index.Value = decimal.Decimal(data[4])
index["open"] = float(data[1])
index["high"] = float(data[2])
index["low"] = float(data[3])
index["close"] = float(data[4])
index["adj_close"] = float(data[5])
index["volume"] = int(data[6])
except Exception as e:
print(f"Error: {line}")
print(f"Exception: {e}")
return None

+ Expand

Mia Alissi INVESTOR

December 2023

The error message "Index was outside the bounds of the array" typically occurs when you're trying to access an element at an index that doesn't exist in the array or list. In your case, it seems like the error is occurring when you're trying to split the line of data and assign it to different variables.

In your Reader method, you're splitting the line of data into a list with data = line.split(','). Then, you're trying to convert this list into a DataFrame with data = self.handle_nulls(pd.DataFrame(data[1:], columns=["open", "high", "low", "close", "adj_close", "volume"])).

The problem here is that line.split(',') returns a list, and you're trying to convert this list into a DataFrame, which expects a 2D array-like structure. This is likely causing the "Index was outside the bounds of the array" error.

Instead of converting the list into a DataFrame, you should directly assign the elements of the list to the corresponding variables. Here's how you can do it:

try:
    data = line.split(',')
    if data[0] == "Date":  
        return None
    index.Time = datetime.strptime(data[0], "%Y-%m-%d")
    index.EndTime = index.Time + timedelta(days=1)
    index.Value = decimal.Decimal(data[4])  
    index["open"] = float(data[1])
    index["high"] = float(data[2])
    index["low"] = float(data[3])
    index["close"] = float(data[4])
    index["adj_close"] = float(data[5])
    index["volume"] = int(data[6])
except Exception as e:          
    print(f"Error: {line}")
    print(f"Exception: {e}")
    return None
+ Expand
- Collapse

Please note that the indices in data might need to be adjusted depending on the actual structure of your data.

Upvote

AleNoc

Alessandro Nocentini | December 2023

don't solve the problem

Mia Alissi

Mia Alissi | December 2023

Based on the information provided in the conversation, it seems like the error "Index was outside the bounds of the array" is occurring due to an attempt to access an element at an index that doesn't exist in the array or list. This is likely happening when the line of data is being split and assigned to different variables in the Reader method of the UcitsEtf class.

The issue seems to be with the following line of code:

data = self.handle_nulls(pd.DataFrame(data[1:], columns=["open", "high", "low", "close", "adj_close", "volume"]))

Here, line.split(',') returns a list, and then there's an attempt to convert this list into a DataFrame, which expects a 2D array-like structure. This is likely causing the "Index was outside the bounds of the array" error.

Instead of converting the list into a DataFrame, the elements of the list should be directly assigned to the corresponding variables. Here's how you can do it:

Platform

Radically Open-Source Algorithmic Trading Engine

Join Our Discord Channel

Quarterly Open-Source Trading Competition

Draft Discussions

Bookmarked Discussions

SEARCH DISCUSSIONS

376,300 Quants.

VOTE FOR UPCOMING FEATURES

JOIN OUR Community MAILING LIST

IN THIS RESEARCH

PARTICIPANTS

Actions

Join QuantConnect for Free