Subject: Issues Loading TensorFlow Model (.h5) from Object Store in QuantConnect

Hello Adam,

It seems like you're encountering an issue with loading a TensorFlow model from the Object Store in QuantConnect. This could be due to the way TensorFlow handles .h5 files in the QuantConnect environment.

One possible solution is to save and load your model weights in JSON format instead of .h5. This approach has been used successfully in the QuantConnect community. Here's an example of how you can do this:

# Save model weights to JSON
weights_list = model.get_weights()
weights_json = json.dumps(weights_list, cls=NumpyArrayEncoder)
self.ObjectStore.Save('weights', weights_json)

# Load model weights from JSON
weights_json = self.ObjectStore.Read('weights')
weights_list = json.loads(weights_json)
model.set_weights(weights_list)

In this example, NumpyArrayEncoder is a custom JSON encoder for numpy arrays:

class NumpyArrayEncoder(JSONEncoder):
    def default(self, obj):
        if isinstance(obj, np.ndarray):
            return obj.tolist()
        return JSONEncoder.default(self, obj)

Another approach is to serialize the TensorFlow graph and weights to JSON format, then save these in the ObjectStore by using the Save method. Here's an example of how you can do this:

graph_definition = tf.compat.v1.train.export_meta_graph()
json_graph = json_format.MessageToJson(graph_definition)

def get_json_weights(sess):
    weights = sess.run(tf.compat.v1.trainable_variables())
    weights = [w.tolist() for w in weights]
    weights_list = json5.dumps(weights)
    return weights_list

json_weights = get_json_weights(sess)
sess.close()

qb.ObjectStore.Save('graph', json_graph)
qb.ObjectStore.Save('weights', json_weights)

You can then load the model from the Object Store in your algorithm:

def initialize(self) -> None:
    model_key = 'model.keras'
    if self.object_store.contains_key(model_key):
        file_name = self.object_store.get_file_path(model_key)
        self.model = tf.keras.models.load_model(file_name)

Please note that you need to have storage create permissions to save data in the Object Store if you run algorithms in QuantConnect Cloud.

I hope this helps! Let me know if you have any other questions.

Sources:

• QuantConnect Forum Discussion: Loading TensorFlow Model Trained Off-Site
• QuantConnect Research: Predicting Future Prices With TensorFlow
• QuantConnect Documentation: TensorFlow
• QuantConnect Documentation: Object Store