Help Requested! PyTorch LSTM Model

I have resolve the issues by rewriting the code to as follows:

# Load and preprocess the data
qb = QuantBook()
spy = qb.AddEquity("SPY")
history = qb.History(qb.Securities.Keys, datetime(2020, 1, 1), datetime(2023, 1, 1), Resolution.Daily)

# Select the close column and then call the unstack method.
close = history['close'].unstack(level=0)

# Initialize MinMaxScaler to scale the data onto [0,1].
scaler = MinMaxScaler(feature_range=(0, 1))
df = pd.DataFrame(scaler.fit_transform(close), index=close.index)

# Select input data
input_ = df.iloc[1:]

# Shift the data for 1-step backward as training output result.
output = df.shift(-1).iloc[:-1]

# Split the data into training and testing sets.
splitter = int(input_.shape[0] * 0.8)
X_train = input_.iloc[:splitter]
X_test = input_.iloc[splitter:]
y_train = output.iloc[:splitter]
y_test = output.iloc[splitter:]

# training
train_features = []
train_labels = []
for i in range(10, X_train.shape[0]):
   train_features.append(X_train.iloc[i-10:i].values.reshape(-1, 1))
   train_labels.append(y_train.iloc[i])
train_features, train_labels = np.array(train_features), np.array(train_labels)
train_features = np.reshape(train_features, (train_features.shape[0], train_features.shape[1], 1))

# testing
test_features = []
test_labels = []
for i in range(10, X_test.shape[0]):
   test_features.append(X_test.iloc[i-10:i].values.reshape(-1, 1))
   test_labels.append(y_test.iloc[i])
test_features, test_labels = np.array(test_features), np.array(test_labels)
test_features = np.reshape(test_features, (test_features.shape[0], test_features.shape[1], 1))

# Convert the data to PyTorch tensors
X_train = torch.from_numpy(train_features).float()
y_train = torch.from_numpy(train_labels).float()
X_test = torch.from_numpy(test_features).float()
y_test = torch.from_numpy(test_labels).float()

Glad to see you figured it out. FYI, the reason behind the error is because LSTM/RNN models accept input tensors of shape (N, T, K), where N is the number of sequences, T are timesteps in each sequence, and K is the dimensionality of the features.

Since you mentioned that you are planning on using other features, be careful with some of the reshape logic in your updated code as it won't work as expected with more than 1 feature. To generalize your code to multiple features, simplest way to reshape it properly is to do something like

df = # 2d DataFrame, shape (n, K)
T = 10
X = []
for i in range(T, df.shape[0]):
    x_i = df.iloc[(i-T):i, :].values   # Previous 10 values of features, shape (T, K)
    x_i = np.expand_dims(x_i, axis=0)  # Add a dimension, shape (1, T, K)
    X.append( x_i )
X = np.concatenate(X, axis=0)  # Shape (N, T, K), can then use as inputs to RNN models

Batching just means that during training you will pass in batches of sequences in each training step, i.e. split N into some smaller batches, for stochastic gradient descent. I would also suggest starting with a simpler model (e.g. feedforward neural network, random forest, etc) if you aren't that familiar with RNNs as they have some nuances, especially for financial data.