Serialising Machine Learning Objects

Good afternoon!

Yes, it is possible to save and load models, including Neural Networks, Random Forests, and Gradient Boosters, in the Object Store in QuantConnect. The Object Store is a key-value storage location that allows you to save and retrieve data in QuantConnect's cache. It is shared across the entire organization, and using the same key, you can access data across all projects in an organization.

For Scikit-Learn models, you can save models in the Object Store by following these steps:

1. Set the key name of the model to be stored in the Object Store.

   model_key = "model"
   

2. Call the GetFilePath method with the key.

   file_name = qb.ObjectStore.GetFilePath(model_key)
   

3. Call the dump method with the model and file path.

   joblib.dump(model, file_name)
   

   If you dump the model using the joblib module before you save the model, you don't need to retrain the model.

For Keras models, you can save models in the Object Store by following these steps:

1. Set the key name of the model to be stored in the Object Store.

   model_key = "model"
   

2. Call the GetFilePath method with the key.

   file_name = qb.ObjectStore.GetFilePath(model_key)
   

3. Call the save method with the file path.

   model.save(file_name)
   

To load the models, you can follow these steps:

1. Call the ContainsKey method.

   qb.ObjectStore.ContainsKey(model_key)
   

2. Call the GetFilePath method with the key.

   file_name = qb.ObjectStore.GetFilePath(model_key)
   

3. Call the load method with the file path, environment, and policy (for Stable Baselines).

   loaded_model = DQN.load(file_name, env=env, policy=MlpPolicy)
   

For TensorFlow models, you might need to serialize the model weights and architecture into a format that can be stored in the Object Store, similar to how it's done with PyTorch models as discussed in the forum. However, the exact steps might vary depending on the specifics of your TensorFlow model.

Please note that the process of adding new libraries takes 2-4 weeks to complete as they need to be reviewed for security and deployment.

Sources:

• QuantConnect Documentation: Scikit-Learn
• QuantConnect Documentation: Keras
• QuantConnect Documentation: Stable Baselines
• QuantConnect Forum Discussion

Thank you, Mia! What about if I wanted to save the model as part of a larger object and store that in the ObjectStore? Like, for example, what if I wanted to have:

ModelData:

• ModelData.Structure is the model's serialised structure
• ModelData.Name is the name of the model
• ModelData.Time is the time the model took to train
• …

I was thinking we could use the dictionary-saving method you created in this question, but your solution saves the model directly. Could we instead store it in an intermediate string variable?

Thanks a lot!

Yes, you can certainly save the model as part of a larger object and store that in the Object Store. You can create a dictionary to hold the model's serialized structure, name, and training time, and then save this dictionary to the Object Store.

Here's an example of how you might do this:

import joblib
import base64
import pickle

# Assuming 'model' is your trained model
serialized_model = joblib.dump(model)
encoded_model = base64.b64encode(serialized_model)

model_data = {
    'Structure': encoded_model,
    'Name': 'My Model',
    'Time': '10 minutes'  # replace with actual training time
}

# Convert the dictionary to a string
model_data_str = pickle.dumps(model_data)

# Save the string to the Object Store
model_key = 'model_data'
qb.ObjectStore.Save(model_key, model_data_str)

In this example, we first serialize the model using joblib.dump, then encode the serialized model to a base64 string so it can be stored in a dictionary. We then convert the dictionary to a string using pickle.dumps before saving it to the Object Store.

To retrieve the model data from the Object Store, you can do:

# Load the string from the Object Store
model_data_str = qb.ObjectStore.ReadBytes(model_key)

# Convert the string back to a dictionary
model_data = pickle.loads(model_data_str)

# Decode and deserialize the model
encoded_model = model_data['Structure']
serialized_model = base64.b64decode(encoded_model)
model = joblib.load(serialized_model)

Please note that this is a simplified example and you might need to adjust the code to suit your specific needs. For instance, you might need to handle errors and edge cases, and you might want to use a more efficient serialization format if your model is large.

Also, please note that the Object Store has helper methods to store strings, JSON objects, XML objects, and bytes. If you need to store an object that is in a different format, you need to encode it to one of the supported data types.

Sources:

• QuantConnect Documentation: Object Store
• QuantConnect Documentation: Scikit-Learn