Share New Research 
Hello
my stategy need to store status in an object to guarantee persistance across redeployments and potential crashes
class statusPair:
def __init__(self):
self.positionDay = ''
self.block_long = False
self.block_short = False
I've tryed to store it in object store using SaveJson method
algorithm.ObjectStore.SaveJson(self.statusKey, self.statusObj)
where self.statusObj is a statusPair object, according to the documentation usage example:
self.ObjectStore.SaveJson("key", objectValue)but I get the following error:
Runtime Error: Trying to dynamically access a method that does not exist throws a TypeError exception. To prevent the exception, ensure each parameter type matches those required by the SaveJson method. Please checkout the API documentation. do I need to "prepare" my object before using the saveJson Method?could you provide an example of best practice of how to do it ?
what would be the best options among the various methods available for storing objects in objectstore?
a more detailed guide ont this topic would be of great help!
thanks
Adam W
> do I need to "prepare" my object before using the saveJson Method?
Yes you need to first serialize the object into a compatible format. JSON works only on certain data types in Python, and (besides hacky ways of encoding your custom object statusPair into JSON-compatible) you are probably better off just using pickle.
This is done by:
self.statusObj = statusPair() # Your custom object # Serialization into bytes import pickle serializedString = pickle.dumps(self.statusObj) # Save into ObjectStore self.ObjectStore.Save("MyObject", serializedString) or self.ObjectStore.SaveBytes("MyObject", serializedString) ## # Load from ObjectStore deserializedString = self.ObjectStore.Read("MyObject") loadStatusObj = pickle.loads(self.statusObj) assert loadStatusObj == self.statusObjAdam W
Typo:
pickle.loads(self.statusObj) # Ignore this pickle.loads(deserializedString) # This onePier Paolo
Hello Adam,
I've tryed to create a script that uses pickle for encoding as per your suggestion
but unfortunately doesn't seem to work for me with both Save and Savebytes
here's the code
from dateutil import parser import pickle class statusObj: def __init__(self): self.invested = False self.positionDay = '2020-01-01' class ResistanceUncoupledThrustAssembly(QCAlgorithm): def Initialize(self): self.SetStartDate(2020, 8, 13) # Set Start Date self.SetCash(100000) # Set Strategy Cash self.spy = self.AddEquity('SPY', Resolution.Daily).Symbol if self.ObjectStore.ContainsKey("MyObject"): self.ObjectStore.Delete("MyObject") self.status=statusObj() def OnData(self, data): if not self.Portfolio.Invested: self.SetHoldings(self.spy, 1) self.status.invested=True self.status.positionDay=str(self.Time.day) self.Log(str(self.status.positionDay)) serialized=pickle.dumps(self.status) self.ObjectStore.SaveBytes("MyObject", serialized) else: if self.ObjectStore.ContainsKey("MyObject"): deserialized = self.ObjectStore.ReadBytes("MyObject") storedStatus = pickle.loads(deserialized) assert storedStatus==self.status self.Log(str(storedStatus.positionDay)) date = parser.parse(storedStatus.PositionDay) self.Liquidate(self.spy) self.status.invested=False self.positionDay = '2020-01-01'Pier Paolo
the Save method does not accept the encoded argument, while saveBytes accept the argoument but when I try to retreive the object from store It gets a Byte[] and it seems it's not what is expected to restore the object back..
Pier Paolo
Shile l've tried to use the save method in the posted script instead of SaveBytes but it doesn't work. How should I modify it in order to make it work? Could you provide an example?
Aaron Janeiro Stone
From what I gather, you are saying that the file saves correctly but you are unable to get python to read the returned byte object (that is, self.ObjectStore.ReadBytes("MyObject") is of form Byte[])? I'd try it right now if my nodes weren't all busy, but give the following a read:
https://stackoverflow.com/questions/37789281/passing-bytes-as-parameter-to-c. Basically, your stored item is coming back as a .NET object (System.Byte, unlike python's bytes object). Now, translate into something python can understand. I'd guess deserialized=bytes(self.ObjectStore.ReadBytes("MyObject")).
Pier Paolo
Great spot Aaron,
it finally does the job!
ps: the assert test still fails, but values get stored correctly
Lars Temme
This is VERY useful. I will use this code for production live trading. Creating a large dictionary for the stateful metadata. Tracking trade friction, high and low prices from tick streams. Just need to make sure that in love trading and deployed code we use unique object store keys.
Grokpot
For those who come here wondering how to save and load a pytorch model in quantconnect from notebook to backtesting:
In your notebook:
model_key = 'mykey' # https://stackoverflow.com/questions/42703500/best-way-to-save-a-trained-model-in-pytorch # https://pytorch.org/docs/master/notes/serialization.html buffer = io.BytesIO() torch.save(model.state_dict(), buffer) buffer.seek(0) qb.ObjectStore.SaveBytes(model_key, buffer.read())In your backtesting file:
# If you're using a custom Module you have to copy it into your backtesting file, # otherwise torch.load won't recognize the Module type class GRU(nn.Module): def __init__(self, input_dim, hidden_dim, num_layers, output_dim): ... def forward(self, x): ... def Initialize(self): model_key = 'mykey' if self.ObjectStore.ContainsKey(model_key): modelBytes = self.ObjectStore.ReadBytes(model_key) # ReadBytes returns a C# Bytes[] object which must be cast to python bytes buffer = io.BytesIO(bytes(modelBytes)) model = torch.load_state_dict(buffer)Derek Melchin
Hi Grokpot,
Thanks for sharing this solution with the community!
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Grokpot
EDIT to my above post:
Notebook:
# FYI: When the model export becomes too big (9MB), a "key not found in objectstore" error appears import io model_key = 'pytorch-3' # https://stackoverflow.com/questions/42703500/best-way-to-save-a-trained-model-in-pytorch # https://pytorch.org/docs/master/notes/serialization.html buffer = io.BytesIO() torch.save(model.state_dict(), buffer) buffer.seek(0) qb.ObjectStore.SaveBytes(model_key, buffer.read())main.py:
# If you're using a custom Module you have to copy it into your backtesting file, # otherwise torch.load won't recognize the Module type class GRU(nn.Module): def __init__(self, input_dim, hidden_dim, num_layers, output_dim): ... def forward(self, x): ... def Initialize(self): model_key = 'pytorch-3' modelBytes = self.ObjectStore.ReadBytes(model_key) # ReadBytes returns a C# Bytes[] object which must be cast to python bytes buffer = io.BytesIO(bytes(modelBytes)) self.model = GRU(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers) self.model.load_state_dict(torch.load(buffer)) # self.model.eval() # Sometimes needed depending on the modelPier Paolo
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!