Hi! does some has maybe a working code how to download .H5 TF keras model file from dropbox to QC? I know there is the download method in python which seems to work in QC to download e.g. CSV files. But how to use it to download binary files like H5 and additionally, how to make it working in kera where the open file method just excpet to get the file name?
I know the save/open file is still a huge limitation in QC, but at least the models could be loaded from dropbox in the moment until a new file open/save option is merged?
Thanks so much for any help!
Rob Anderson
Did you ever find a way to do this? I'd also like to import a trained H5 model into QC.
Rob Anderson
While not a solution for working with H5 files, I have found a way by to work with external keras models by just importing the saved model weights in json format from DropBox. You need to recreate the model and load the weights in QC, but it works as expected.
Varad Kabade
Hi Rob Anderson,
Thank you for your contribution. Would you be interested in sharing your implementation with the community?
Best,
Varad Kabade
Michael Dingens
Hi Rob Anderson,
I am also interested in your solution. Right now all my models have been trained and stored as h5 files locally. So I wonder if I can just load them again locally, then use the Model.to_json(**kwargs) method which is mentioned here to save the model weights as json and then upload and load the model weights as json file to QC with tf.keras.models.model_from_json(json_string, custom_objects=None) as described in the same link.Â
As far as I understand, I need to copy paste the model architecture and compile the model with these pretrained weights in QC again to get the same model, right? Does that work with the Download function at the moment?
Michael Dingens
Maybe this here works. Will try it:
Michael Dingens
Ok, after some searching, I've read that the internal Keras to_json only exports the model but not the weights.
So for weights, I came across the following solution that copies the weights to numpy arrays or lists and then dumps them to files.
Rob Anderson
HI Michael and Varad,
I'll try to find the time to load a more complete answer, but in the meanwhile, here are the essential steps I followed. Hope this helps 😊
Step 1 - save model weights (Notebook)
I compiled and ran my model in Jupyter Notebook, e.g. essential code/steps…
Step 2 - upload weights to Dropbox
Upload weights.json to Dropbox folder and copy Dropbox link.
Step s - load model weights (QC)
Here is an extract of workable code in my main.py file.
Notes:
Â
Rob Anderson
Sorry, the indentation got screwed up when saving my code examples… hopefully you can recreate successfully.
Michael Dingens
Thank you, Rob! Great work!
I managed to load the weights and get my LSTM predictions in Lean CLI local installation last night. Somehow I ran into issues with the Dropbox solution for online backtesting. It took the html code instead of the json. So your code is really appreciated and I'll check what mistake I did.
That's how I managed to save and load the weights locally:
Â
and in the callbacks:
Â
2. In the Lean Python project, after creating the model structure:
So now the LSTM is making predictions in the Lean CLI local backtest when it receives the LSTM windows. I just still need to save and load the scaler, probably in a similar way.
Rob Anderson
Hi Michael,
That's great news, glad to hear you're making progress!
The most likely cause of HTML being returned from Dropbox is not changing the last character in the URL to 1 to signal a download.
I like your implementation and will look at using your class for encoding (my previous attempts weren't all that successful).Â
Regarding scaling, I have saved min/max values for series used during training to a separate json file and imported them in a similar way.
Cheers, RobÂ
Carpediem911
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!