I get the error when fitting an LSTM using keras/pandas. I'm not sure what I can do to fix it.
import random
import tensorflow as tf
from keras.models import Sequential
from keras.layers import LSTM, Dense
from keras.preprocessing.sequence import TimeseriesGenerator
from sklearn.model_selection import train_test_split
# Get data
qb = QuantBook()
eq = qb.AddEquity("SPY")
data = qb.History(eq.Symbol, timedelta(days=30), Resolution.Daily).droplevel("symbol")
# Reset randomization seeds
np.random.seed(0)
tf.random.set_random_seed(0)
random.seed(0)
timesteps = 1
batch_size = 32
# Just using these features as an example
X = data[['low', 'high']][:-1]
Y = data['close'].shift(-1)[:-1]
X_train, _X, y_train, _y = train_test_split(X, Y, test_size=0.5, shuffle=False, random_state=1)
X_validation, X_test, y_validation, y_test = train_test_split(_X, _y, test_size=0.5, shuffle=False, random_state=1)
train = TimeseriesGenerator(X_train, y_train, timesteps, batch_size=batch_size)
validation = TimeseriesGenerator(X_validation, y_validation, timesteps, batch_size=batch_size)
test = TimeseriesGenerator(X_test, y_test, timesteps, batch_size=batch_size)
model = Sequential(name='Example Model')
model.add(LSTM(50, name='LSTM', stateful=True, batch_input_shape=(batch_size, timesteps, X_train.shape[1])))
model.add(Dense(1, name='Output'))
model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam', sample_weight_mode='temporal')
print(model.summary())
model.fit_generator(train, epochs=10, validation_data=validation, verbose=1)
loss, accuracy = model.evaluate_generator(test, verbose=0)
print(f'Model Accuracy: {accuracy}')
Full backtrace:
Using TensorFlow backend.
WARNING:tensorflow:From /opt/miniconda3/lib/python3.6/site-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
Model: "Example Model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
LSTM (LSTM) (32, 50) 10600
_________________________________________________________________
Output (Dense) (32, 1) 51
=================================================================
Total params: 10,651
Trainable params: 10,651
Non-trainable params: 0
_________________________________________________________________
None
WARNING:tensorflow:From /opt/miniconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
Epoch 1/10
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/opt/miniconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
2896 try:
-> 2897 return self._engine.get_loc(key)
2898 except KeyError:
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 0
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
~/PandasMapper.py in wrapped_function(*args, **kwargs)
72 try:
---> 73 return f(*args, **kwargs)
74 except KeyError as e:
/opt/miniconda3/lib/python3.6/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
2898 except KeyError:
-> 2899 return self._engine.get_loc(self._maybe_cast_indexer(key))
2900 indexer = self.get_indexer([key], method=method, tolerance=tolerance)
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()
KeyError: 0
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
~/PandasMapper.py in wrapped_function(*args, **kwargs)
72 try:
---> 73 return f(*args, **kwargs)
74 except KeyError as e:
/opt/miniconda3/lib/python3.6/site-packages/pandas/core/frame.py in __getitem__(self, key)
2994 return self._getitem_multilevel(key)
-> 2995 indexer = self.columns.get_loc(key)
2996 if is_integer(indexer):
~/PandasMapper.py in wrapped_function(*args, **kwargs)
75 oKey = [arg for arg in args if isinstance(arg, str)]
---> 76 raise KeyError(f"No key found for either mapped or original key. Mapped Key: {mKey}; Original Key: {oKey}")
77
KeyError: 'No key found for either mapped or original key. Mapped Key: []; Original Key: []'
During handling of the above exception, another exception occurred:
KeyError Traceback (most recent call last)
<ipython-input-2-290f7cfdc25a> in <module>
30 model.compile(loss='logcosh', metrics=['mean_absolute_percentage_error'], optimizer='Adam', sample_weight_mode='temporal')
31 print(model.summary())
---> 32 model.fit_generator(train, epochs=10, validation_data=validation, verbose=1)
33 loss, accuracy = model.evaluate_generator(test, verbose=0)
34 print(f'Model Accuracy: {accuracy}')
/opt/miniconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py in wrapper(*args, **kwargs)
89 warnings.warn('Update your `' + object_name + '` call to the ' +
90 'Keras 2 API: ' + signature, stacklevel=2)
---> 91 return func(*args, **kwargs)
92 wrapper._original_function = func
93 return wrapper
/opt/miniconda3/lib/python3.6/site-packages/keras/engine/training.py in fit_generator(self, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
1730 use_multiprocessing=use_multiprocessing,
1731 shuffle=shuffle,
-> 1732 initial_epoch=initial_epoch)
1733
1734 @interfaces.legacy_generator_methods_support
/opt/miniconda3/lib/python3.6/site-packages/keras/engine/training_generator.py in fit_generator(model, generator, steps_per_epoch, epochs, verbose, callbacks, validation_data, validation_steps, validation_freq, class_weight, max_queue_size, workers, use_multiprocessing, shuffle, initial_epoch)
183 batch_index = 0
184 while steps_done < steps_per_epoch:
--> 185 generator_output = next(output_generator)
186
187 if not hasattr(generator_output, '__len__'):
/opt/miniconda3/lib/python3.6/site-packages/keras/utils/data_utils.py in get(self)
623 except Exception:
624 self.stop()
--> 625 six.reraise(*sys.exc_info())
626
627
/opt/miniconda3/lib/python3.6/site-packages/six.py in reraise(tp, value, tb)
717 if value.__traceback__ is not tb:
718 raise value.with_traceback(tb)
--> 719 raise value
720 finally:
721 value = None
/opt/miniconda3/lib/python3.6/site-packages/keras/utils/data_utils.py in get(self)
608 try:
609 future = self.queue.get(block=True)
--> 610 inputs = future.get(timeout=30)
611 except mp.TimeoutError:
612 idx = future.idx
/opt/miniconda3/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
642 return self._value
643 else:
--> 644 raise self._value
645
646 def _set(self, i, obj):
/opt/miniconda3/lib/python3.6/multiprocessing/pool.py in worker(inqueue, outqueue, initializer, initargs, maxtasks, wrap_exception)
117 job, i, func, args, kwds = task
118 try:
--> 119 result = (True, func(*args, **kwds))
120 except Exception as e:
121 if wrap_exception and func is not _helper_reraises_exception:
/opt/miniconda3/lib/python3.6/site-packages/keras/utils/data_utils.py in get_index(uid, i)
404 The value at index `i`.
405 """
--> 406 return _SHARED_SEQUENCES[uid][i]
407
408
/opt/miniconda3/lib/python3.6/site-packages/keras_preprocessing/sequence.py in __getitem__(self, index)
371
372 samples = np.array([self.data[row - self.length:row:self.sampling_rate]
--> 373 for row in rows])
374 targets = np.array([self.targets[row] for row in rows])
375
~/PandasMapper.py in wrapped_function(*args, **kwargs)
74 except KeyError as e:
75 oKey = [arg for arg in args if isinstance(arg, str)]
---> 76 raise KeyError(f"No key found for either mapped or original key. Mapped Key: {mKey}; Original Key: {oKey}")
77
78 wrapped_function.__name__ = f.__name__
KeyError: 'No key found for either mapped or original key. Mapped Key: []; Original Key: []'
Alexandre Catarino
Hi Jake MitchellÂ
Thank you for the report.
We have created a GitHub issue to address the problem:
PandasMapper Subset Exception #5950
Please subscribe for updates.
Meanwhile, you might be able to work around it by creating a new data frame with the data from the original one. It will remove the PandasMapper enhancement.
Best regards,
Alex
Jake Mitchell
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!