Share New Research 
Hey all!
I'm here to publicly announce a feature that we've all been waiting for, history and warmup! With these new features you can request historical data from any point in the algorithm. You can also specify a warm up period where we'll pump historical data through the algorithm to get it into a ready state. This is great for live trading when you have some indicators that may take weeks, months, or even years to reach a ready state. Now that same algorithm can be ready to start trading within minutes of deploying!
Here's a link to all the History and Warmup API methods on the base algorithm class.
Have a look at the simple history demo algorithm I put together for you guys. Feel free to ask questions!
Stephen Oehler
Ah I see. Could this not be done within the CustomSecurityInitializer? Or does each individual security in the universe get initialized over and over again with each universe rotation?
Stephen Oehler
I was thinking something like this, where the code is copy-pasted from the EMA cross example algorithm. All I've added are lines 37-42. But I'm not certain this would work, as I'm given the error: "An object reference is required for the non-static field, method, or property, QCAlgorithm.History(Symbol, int, Resolution)." I'm guess this is because SelectionData in this example has no interface with QCAlgorithm?
using System; using System.Collections.Concurrent; using System.Linq; using QuantConnect.Data.Market; using QuantConnect.Data.UniverseSelection; using QuantConnect.Indicators; namespace QuantConnect { ///
/// In this algorithm we demonstrate how to perform some technical analysis as
/// part of your coarse fundamental universe selection
///
public class EmaCrossUniverseSelectionAlgorithm : QCAlgorithm
{
// tolerance to prevent bouncing
const decimal Tolerance = 0.01m;
private const int Count = 10;
// use Buffer+Count to leave a little in cash
private const decimal TargetPercent = 0.1m;
private SecurityChanges _changes = SecurityChanges.None;
// holds our coarse fundamental indicators by symbol
private readonly ConcurrentDictionary _averages = new ConcurrentDictionary();
// class used to improve readability of the coarse selection function
private class SelectionData
{
public readonly ExponentialMovingAverage Fast;
public readonly ExponentialMovingAverage Slow;
public SelectionData(string symbol)
{
Fast = new ExponentialMovingAverage(100);
Slow = new ExponentialMovingAverage(300);
var history = History(symbol, 350);
foreach (var tradeBar in history)
{
Fast.Update(tradeBar.EndTime, tradeBar.Close);
Slow.Update(tradeBar.EndTime, tradeBar.Close);
}
}
// computes an object score of how much large the fast is than the slow
public decimal ScaledDelta
{
get { return (Fast - Slow)/((Fast + Slow)/2m); }
}
// updates the EMA50 and EMA100 indicators, returning true when they're both ready
public bool Update(DateTime time, decimal value)
{
return Fast.Update(time, value) && Slow.Update(time, value);
}
}
///
/// Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.
///
public override void Initialize()
{
UniverseSettings.Leverage = 2.0m;
UniverseSettings.Resolution = Resolution.Daily;
SetStartDate(2010, 01, 01);
SetEndDate(2012, 01, 01);
SetCash(100*1000);
SetBrokerageModel(BrokerageName.TradierBrokerage);
SetSecurityInitializer(new CustomSecurityInitializer(BrokerageModel, DataNormalizationMode.Raw));
AddUniverse(coarse =>
{
return (from cf in coarse
// grab th SelectionData instance for this symbol
let avg = _averages.GetOrAdd(cf.Symbol, sym => new SelectionData(cf.Symbol))
// Update returns true when the indicators are ready, so don't accept until they are
where avg.Update(cf.EndTime, cf.Price)
// only pick symbols who have their 50 day ema over their 100 day ema
where avg.Fast > avg.Slow*(1 + Tolerance)
// prefer symbols with a larger delta by percentage between the two averages
orderby avg.ScaledDelta descending
// we only need to return the symbol and return 'Count' symbols
select cf.Symbol).Take(Count);
});
}
///
/// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
///
/// TradeBars dictionary object keyed by symbol containing the stock data
public void OnData(TradeBars data)
{
if (_changes == SecurityChanges.None) return;
// liquidate securities removed from our universe
foreach (var security in _changes.RemovedSecurities)
{
if (security.Invested)
{
Liquidate(security.Symbol);
}
}
// we'll simply go long each security we added to the universe
foreach (var security in _changes.AddedSecurities)
{
SetHoldings(security.Symbol, TargetPercent);
}
}
///
/// Event fired each time the we add/remove securities from the data feed
///
/// Object containing AddedSecurities and RemovedSecurities
public override void OnSecuritiesChanged(SecurityChanges changes)
{
_changes = changes;
}
}
} Michael Handschuh
A very important point here is that data that comes through the universe selection functions may not have corresponding security objects. That is to say, it's just the data accompanied by a symbol which defines the security's identifier, but not the security itself (including models, price, ect...). So a custom ISecurityInitializer is great for modifying a security once it's been selected by the universe, but won't do anything for data with no matching security in the universe selection data. Another concern with doing these types of history calls one at a time is it can take a long time, so it would be best to collect all of the symbols and make a request on the first selection.
Michael Handschuh
RE: Error The error you're receiving is because the History function is a method on your algorithm instance and you're trying to invoke it from another class. You could modify the constructor to accept an instance of QCAlgorithm and then call algorithm.History(symbol, 350);. Here's the updated constructor:
EDIT: While this should work as we want it to, it may be very slow on the first coarse selection since it will be making about ~7000 unique history requests.public SelectionData(QCAlgorithm algorithm, string symbol) { Fast = new ExponentialMovingAverage(100); Slow = new ExponentialMovingAverage(300); var history = algorithm.History(symbol, 350); foreach (var tradeBar in history) { Fast.Update(tradeBar.EndTime, tradeBar.Close); Slow.Update(tradeBar.EndTime, tradeBar.Close); } }Stephen Oehler
Gotcha, thanks very much for the help. So I attempted the above with the following code (where, for the QCAlgorithm object, I pass in "this" within the coarse universe Linq statement), but it ended up not making any trades (flatlining) throughout the entire trade period. If it's not too much to ask, could i perhaps email you the pid to investigate when you have the time? I'm unfortunately unable to request log output of the universe to debug on my end, as it would overload your system quite quickly! :-)
public SelectionData(QCAlgorithm algorithm, string symbol, int window) { SMAMetric = new SimpleMovingAverage(window); RollingMetric = new RollingWindow(window);
var history = algorithm.History(symbol, 21, Resolution.Daily); // 21 days
foreach (var tradeBar in history)
{
Update(tradeBar.EndTime, tradeBar.Close); // Method within SelectionData class to update SMAMetric
}
} Michael Handschuh
I recommend trying to run a very simple algorithm when trying new things out. I would start a new project and paste in the SelectionData class. Use a very simple coarse selector, just to prove the SelectionData part is working. The key to debugging systems like this is to work in small pieces :) If you can't get the trimmed down algorithm to work, post it here and I'll take a look. In the meantime, you can email me the pid and I'll see if I can take a look at it this evening for you.
Stephen Oehler
Great idea :-) I'll attempt a simpler study before I bother you with a PID
Stephen Oehler
Hey Michael, Just a quick update; I managed to find some time to take a look into this further. Unfortunately I'm not so bright at this sort of thing, so I apologize in advance. Attached is a "pared down" coarse universe algorithm that should (I think) log the initialization of every symbol, then log each time that symbol was updated via history. So far, it's returning the following line each day for all four months of the backtest:
Initializing: A RPTMYV3VC57PSo two problems I see: 1.) The symbol is indecipherable, so I'm fairly certain I'm trying to implicitly ".ToString()" the wrong entity 2.) The update isn't logging, which indicates the history is empty. I believe this to be related to point #1, as we could be passing bogus symbol strings to the algorithm to go find history for. Hope I'm not too far off base with these assumptions :-PStephen Oehler
Ok so in the previous example, I was not handling the Symbol very well (passing it around like it was a string, etc.). I corrected it in the code below. Unfortunately, I'm not able to get much further, as Debug/Log statements that are too iterative are blocked from view (and rightly so, I suppose, else it would flood the system). As a result, I can't see what is going on inside. I DO see that the very first asset is being initialized ($A) for every day, but past that point its a guess. It's weird, though; every day it initializes $A. I thought the universe is initialized just once at the first day of the backtest. It looks like it's re-initializing each day. Very confused here...
Stephen Oehler
I also just tried to be a little creative by putting Quit() statements in the SelectionData so I could see at least SOMETHING but no luck. The algorithm completely bypassed the Quit() and kept on running as if it didn't even see it...
Stephen Oehler
I think I figured it out! Michael, do you think you could validate my findings? Attached is the quintessential EMA Cross algorithm with some modifications. I note two things: 1.) The queuing time is indeed a few minutes long because it needs to parse a lot of additional data before starting (as expected). 2.) The algorithm begins trading immediately on the first day (awesome!). Thanks!
Travis Teichelmann
Awesome algorithm Stephen thanks for sharing. Even though the fees are pretty exorbitant those profits are great.
Stephen Oehler
Thanks Travis! It's actually just the EMA Cross example algorithm that they include in the LEAN+ engine, modified to be able to handle history/warmup. Before this point, I don't think there was a "canned" function to handle warmup in a coarse universe. The modifications in this algo should allow you to do that now :-)
RobertKoppel
So i have this little problem using warmup. I record the closing prices with rollingwindow:
if (b.Time.Hour==15 && b.Time.Minute == 59) { Â Â Â history.Add(b); Â Â Â Debug(">>LASTMIN>> " + Time.ToString() +" " + b.Close + " " + history[0].Close + " " + history[1].Close);Â }And the output would be like this: (algo start date 05/05/2016)
>>LASTMIN>> 05/03/2016 16:00:00 16.52500000000 16.52500000000 15.81000000000>>LASTMIN>> 05/04/2016 16:00:00 16.72500000000 16.72500000000 16.52500000000Algorithm finished warming up.>>LASTMIN>> 05/05/2016 09:30:01 16.3900 16.3900 16.72500000000
So it is recording the opening price as closing price, how can this be? On some start dates this does't happen
Jared Broad
Hi Rob, If you're using fill forward data, on second resolution (or there's no trades in the first minute for your security), you'll get the price from yesterday filled forward until the first tick of the day. Alot of stocks have surprising gaps at open where there may be no trades for several seconds or minutes.
Please share an algorithm for us to be able to further assist.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
RobertKoppel
Sorry, bad description of the problem from me. The problem is that right after warmup the
if (b.Time.Hour==15 && b.Time.Minute == 59)statement is ignored and data is recorded at 09:30:01-09:30:59 and on (and i use minute resolution) Any ideas?
RobertKoppel
forgot to attach algo
Thomas
Hi all,
I am struggeling to get my indicators warmed up. I store indicator data in a dictionary where each symbol has the indicator data stored.
Using the SetWarmUp method, the algo is warming up but the indicators are not becoming ready until the time has passed, ie SetWarmUp is not pumping data into my indicators. See the log of attached backtest.
Any suggestions are much appreciated!
Stefano Raggi
Hi Thomas there is nothing wrong with your algorithm, the issue is a bug in LEAN and a fix has been submitted for review here:
Thomas
Thanks Stefano!! SetWarmUp works now as expected
Michael Handschuh
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!