Lean CLI
Object Store
Introduction
The Object Store is an organization-specific key-value storage location to save and retrieve data. Similar to a dictionary or hash table, a key-value store is a storage system that saves and retrieves objects by using keys. A key is a unique string that is associated with a single record in the key-value store and a value is an object being stored. Some common use cases of the Object Store include the following:
- Transporting data between the backtesting environment and the research environment.
- Training machine learning models in the research environment before deploying them to live trading.
The Object Store is shared across the entire organization. Using the same key, you can access data across all projects in an organization.
Local Storage
To add files to your Object Store, add them to the storage directory of your organization workspace or call the Save methods in your algorithm.
To view the contents of the Object Store, run lean object-store ls.
$ lean object-store ls
This command opens File Explorer to show the storage directory. You can delete and rename files in the Object Store directly from the File Explorer. To edit files, open them in a text editor.
Cloud Storage
The CLI enables you to upload files to the Object Store, view a summary of your stored files, and delete files.
Upload Files
To upload files to the Object Store, run lean cloud object-store set <storage-key> <local-file-path>.
$ lean cloud object-store 15737956/signals D:\qc\magic-signals.txt Setting object 15737956/signals in organization d6d62db48592c72e67b534553413b691
If the file fails to upload, you may have insufficient storage space. If you need more, edit your storage plan.
List Directory Contents
To view all of the files and folders in the root directory of the Object Store, run lean cloud object-store ls.
$ lean cloud object-store ls Key Bytes Folder Filename /15710069 None True 15710069 /15727540 None True 15727540 /15727540-1 None True 15727540-1 /15730221 89649 False 15730221 /15730422 89748 False 15730422
To view all of the files and folders that are inside of one of the directories, run lean cloud object-store ls <folder-name>.
$ lean cloud object-store ls 15710069 Key Bytes Folder Filename 15710069/adjusted 60024 False adjusted 15710069/raw 60143 False raw
Get File Metadata
To view the metadata of a file in the Object Store, run lean cloud object-store properties <path/to/file>.
$ lean cloud object-store properties 15710069/adjusted
Bytes Modified Filename Preview
60024 2023-08-30 23:08:23 15710069/adjusted {"12723264
Delete Content
To delete a file or directory in the Object Storage, run lean cloud object-store delete <key>.
$ lean cloud object-store delete 15710069/adjusted
Download Files
Permissioned Institutional clients can build derivative data such as machine learning models and download it from the Object Store. Contact us to unlock this feature for your account.
To download a file or directory from the Object Store, run lean cloud object-store get <key>.
$ lean cloud object-store get 15710069/adjusted Fetching object store download url Unzipping object store keys values into: <Current Directory>
Bulk File Upload
To upload all files from a directory to the Object Store, run the following Python script:
from os import listdir, path
from subprocess import run
SRC = "" # Source directory
DST = "" # Destination directory (empty is root)
if __name__ == '__main__':
command = [ "lean", "cloud", "object-store", "set" ]
for key in listdir(SRC):
fullname = path.join(SRC, key)
if not path.isfile(fullname):
continue
key = DST + '/' + key
args = [key, fullname]
print(' '.join(command + args))
run(command + args)
You can also see our Videos. You can also get in touch with us via Discord.
Did you find this page helpful?
