Machine Learning

Training Models

Introduction

Algorithms usually must process each timeslice within 10 minutes, but the Traintrain method allows you to increase this time to train machine learning models. The length of time you can train depends on your training quotas.

Train Models

To train models immediately, call the Traintrain method and pass in the name of your training method.

# Run an ML model training session and extend method call timeouts.
Train(MyTrainingMethod);
// Run an ML model training session and extend method call timeouts.
self.train(self.my_method)

Immediate training is most useful for training your model when you first deploy your strategy to production or when the model's performance begins to degrade.

Schedule Training Sessions

You can schedule model training sessions in a similar way to a Scheduled Event. To schedule a training session do this, pass in a DateRules and TimeRules argument to the Traintrain method.

// Set TrainingMethod to be executed at 8:00 am every Sunday
Train(DateRules.Every(DayOfWeek.Sunday), TimeRules.At(8, 0), MyTrainingMethod);
# Set TrainingMethod to be executed at 8:00 am every Sunday
self.train(self.date_rules.every(DayOfWeek.SUNDAY), self.time_rules.at(8,0), self.my_training_method)

We recommend you schedule your training sessions for when the market is closed to get the best compute allocation. While the market is open, your CPU is occupied with processing incoming tick data and handling other LEAN events.

Training Quotas

Training resources are allocated with a leaky bucket algorithm where you can use a maximum of n-minutes in a single training session and the number of available minutes refills over time. This design gives you burst allocations when you need them and recharges the allowance to prepare for the next training.

Cloud Quotas

If you execute algorithms in QuantConnect Cloud, see Training Quotas for more information about the training quotas.

Local Quotas

If you execute algorithms locally, the following table shows the default settings for the leaky bucket algorithm:

SettingValue
Capacity (minutes)
120
Time interval (minutes)1440
Refill amount (minutes per time interval)
Capacity / 7

To allow virtually unlimited training for local algorithms, add the following key-value pairs to your Lean / Launcher / config.json file:

"scheduled-event-leaky-bucket-capacity" : 99999999,
"scheduled-event-leaky-bucket-time-interval-minutes" : 1,
"scheduled-event-leaky-bucket-refill-amount": 999999,

Check Model Readiness

In backtests, the Traintrain method is synchronous, so it blocks your algorithm execution while the model trains. In live trading, the Traintrain method is asynchronous, so ensure your model is trained before you continue the algorithm execution. Training occurs on a separate thread, so set a boolean flag to notify your algorithm of the model state. A semaphore is a thread-safe flag you can use to synchronize program operations across different threads. 

# Example of using a flag to signal the model is ready to use:
class SemaphoreTrainingAlgorithm(QCAlgorithm):

    model = None  # Model object
    model_is_training = False  # Model state flag

    def initialize(self) -> None: 
        self.train(self.my_training_method)
    
    def my_training_method(self) -> None: 
        self.model_is_training = True
        # Train the model here.
        self.model_is_training = False
    
    def on_data(self, slice: Slice) -> None: 
        # Don't use the model while it is training.
        if self.model_is_training:
            return
        
        # Once training is complete, use the model.
        result = self.model.predict()

// Example of using a flag to signal the model is ready to use:
public class SemaphoreTrainingAlgorithm : QCAlgorithm
{
    // Initialize the model.
    private MachineLearningModel _model;
    // Initialize the training state flag.
    private bool _modelIsTraining;

    public override void Initialize()
    {
        Train(MyTrainingMethod);
    }

    private void MyTrainingMethod()
    {
        _modelIsTraining = true;
        // Train the model here.
        _modelIsTraining = false;
    }

    public override void OnData(Slice slice)
    {
        // Don't use the model while it is training.
        if (_modelIsTraining)
        {
            return;
        }
        // Once training is complete, use the model.
        var result = _model.Predict();
    }
}

Examples

Demonstration Algorithms
TrainingExampleAlgorithm.py Python TrainingExampleAlgorithm.cs C#

You can also see our Videos. You can also get in touch with us via Discord.

Did you find this page helpful?

Contribute to the documentation: