Share New Research 
Hello Everyone!
We're happy to share we've completed an integration with Tiingo to add their historical news data to QuantConnect's platform! This will enable a new suite of news events and sentiment style strategies able to quickly apply the news to trading!
The engineering behind our new custom data source integrations was pretty epic. We've worked behind the scenes to make it smooth for months - ensuring tickers were properly mapped, dealing with collisions of symbol objects and finally getting new data production ready!
This is an [Alpha Streams Approved] data source. For any alpha streams or competition use, the data is free to consume! For individual live trading, you will need to register for a paid account and get an API token on Tiingo.
With Tiingo you can pull in the news titles, descriptions, tags about the article and other companies mentioned in the news. We're excited to see the awesome new applications possible with the new data!
Check out the example below doing some basic NLP to assign sentiment to words of the news! See more information in the new Alternative Data / Tiingo News Documentation!
Happy Coding,
Jared
Zhiqiang Bi
This looks like a great data source.
Will you also consider adding more text mining packages, such as gensim, fasttext?
A request for these packages has been sent to support.
Jared Broad
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Eugene
great!
Joe Bastulli
Nice!
Jared Broad
@Zhiqiang we merged in the new NLP processing packages. We had an issue with FastText so we couldn't get that in for this round but maybe next month. You can see the full list of supported libraries here.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Zhiqiang Bi
Zhiqiang Bi
Is this data available for Research?
Jared Broad
Yes! Research is now immediately updated with LEAN's latest version; so all new LEAN features are available in Research.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Kieran Flynn
What is the time resolution of news events?
Is it delivered once daily at the close, or like instanteous tick data in backtesting?
Jared Broad
@Kieran the data is "point in time" - streamed to your algorithm as it happens (so its closer to tick data).
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Fireball
Thanks for the addition of this capability! I have attempted to incorporate it in my Alpha, but the algo is running pretty slow with the addition of the Tiingo news. Why is this? Any advice on how to improve the speed? The Tiingo news section starts at line 92.
from clr import AddReference AddReference("System") AddReference("QuantConnect.Algorithm") AddReference("QuantConnect.Indicators") AddReference("QuantConnect.Common") from datetime import datetime, timedelta import numpy as np from System import * from QuantConnect import * from QuantConnect.Data import * from QuantConnect.Algorithm import * from QuantConnect.Indicators import * from System.Collections.Generic import List from QuantConnect.Algorithm.Framework import * from QuantConnect.Algorithm.Framework.Alphas import * from Risk.MaximumSectorExposureRiskManagementModel import MaximumSectorExposureRiskManagementModel from QuantConnect.Data.Custom.Tiingo import * class Compbasealgo(QCAlgorithm): def Initialize(self): '''Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.''' self.SetStartDate(2015, 8, 4) # 5 years up to the submission date self.SetCash(1000000) # Set $1m Strategy Cash to trade significant AUM self.SetBenchmark('SPY') # SPY Benchmark self.SetBrokerageModel(AlphaStreamsBrokerageModel()) self.UniverseSettings.Resolution = Resolution.Daily self.AddAlpha(MixedAlphaModel()) self.SetRiskManagement(MaximumDrawdownPercentPerSecurityCustom(0.01)) self.SetExecution(ImmediateExecutionModel()) self.SetUniverseSelection(LiquidETFUniverse()) self.SetPortfolioConstruction(InsightWeightingPortfolioConstructionModel()) class MixedAlphaModel(AlphaModel): def __init__(self, fastPeriod = 12, slowPeriod = 26, signalPeriod = 9, movingAverageType = MovingAverageType.Simple, resolution = Resolution.Daily): self.fastPeriod = fastPeriod self.slowPeriod = slowPeriod self.signalPeriod = signalPeriod self.movingAverageType = movingAverageType self.resolution = resolution self.insightPeriod = Time.Multiply(Extensions.ToTimeSpan(resolution), fastPeriod) self.bounceThresholdPercent = 0.1 self.symbolData2 = {} resolutionString = Extensions.GetEnumString(resolution, Resolution) movingAverageTypeString = Extensions.GetEnumString(movingAverageType, MovingAverageType) self.Name = '{}({},{},{},{},{})'.format(self.__class__.__name__, fastPeriod, slowPeriod, signalPeriod, movingAverageTypeString, resolutionString) self.direction = InsightDirection.Flat self.newsData = {} self.wordScores = { "bad": -0.5, "good": 0.5, "negative": -0.5, "great": 0.5, "growth": 0.5, "fail": -0.5, "failed": -0.5, "success": 0.5, "nailed": 0.5, "beat": 0.5, "missed": -0.5, "profitable": 0.5, "beneficial": 0.5, "right": 0.5, "positive": 0.5, "large":0.5, "attractive": 0.5, "sound": 0.5, "excellent": 0.5, "wrong": -0.5, "unproductive": -0.5, "lose": -0.5, "missing": -0.5, "mishandled": -0.5, "un_lucrative": -0.5, "up": 0.5, "down": -0.5, "unproductive": -0.5, "poor": -0.5, "wrong": -0.5, "worthwhile": 0.5, "lucrative": 0.5, "solid": 0.5, "beat": 0.5, "missed": -0.5 } self.currentWindow=0.1 self.sentimentMean=0.1 self.symbol2="EURUSD" def Update(self, algorithm, data): insights = [] for key, sd in self.symbolData2.items(): self.symbol=sd.Security.Symbol self.history = algorithm.History(self.symbol, 100, Resolution.Daily) if not self.history.empty: self.historySymbol = str(self.symbol) try: self.symbolData2[self.symbol].WarmUpIndicators2(self.history)#.loc[self.historySymbol]) except KeyError: pass #if sd.Security.Price == 0: # continue tiingoNews=algorithm.Securities[self.symbol].Data.GetAll(TiingoNews) for article in tiingoNews: words = article.Description.lower().split(" ") score = sum([self.wordScores[word] for word in words if word in self.wordScores]) if score == 0.0: continue symbol = article.Symbol.Underlying self.newsData[symbol].Window.Add(score) algorithm.Debug("score "+ str(score)) algorithm.Debug("Count "+ str(self.newsData[symbol].Window.Count)) if self.newsData[symbol].Window.Count < 3: return insights self.currentWindow = self.newsData[symbol].Window[0] self.sentimentMean = sum(self.newsData[symbol].Window)/self.newsData[symbol].Window.Count algorithm.Debug("Sentiment "+ str(symbol)) algorithm.Debug("MACD "+ str(self.symbol)) algorithm.Debug("score "+ str(score)) algorithm.Debug("Count "+ str(self.newsData[symbol].Window.Count)) normalized_signal = sd.MACD.Signal.Current.Value / sd.Security.Price normalized_slope=sd.Reg.Slope.Current.Value/sd.Security.Price algorithm.Debug("MACD "+ str(normalized_signal)) algorithm.Debug("slope "+ str(sd.Reg.Slope.Current.Value)) algorithm.Debug("currentwindow "+ str(self.currentWindow)) algorithm.Debug("sentimentmean "+ str(self.sentimentMean)) #if self.direction == sd.PreviousDirection: # continue if normalized_signal > self.bounceThresholdPercent and normalized_slope>0 and self.currentWindow > (self.sentimentMean*1.1): insight = Insight.Price(sd.Security.Symbol, timedelta(hours = 1), InsightDirection.Up, None, None, None, 0.05) self.direction = InsightDirection.Up insights.append(insight) sd.PreviousDirection = insight.Direction elif normalized_signal < -self.bounceThresholdPercent and normalized_slope<-0 and self.currentWindow < -1: self.direction = InsightDirection.Down insight = Insight.Price(sd.Security.Symbol, timedelta(hours = 1), InsightDirection.Down, None, None, None, 0.05) insights.append(insight) sd.PreviousDirection = insight.Direction elif normalized_signal>-self.bounceThresholdPercent and normalized_signal<self.bounceThresholdPercent: self.direction = InsightDirection.Flat insight = Insight.Price(sd.Security.Symbol, timedelta(hours = 1), InsightDirection.Flat) insights.append(insight) return insights def OnSecuritiesChanged(self, algorithm, changes): for added in changes.AddedSecurities: self.symbol=added.Symbol self.symbolData2[added.Symbol] = SymbolData2(algorithm, added, self.fastPeriod, self.slowPeriod, self.signalPeriod, self.movingAverageType, self.resolution) for removed in changes.RemovedSecurities: data = self.symbolData2.pop(removed.Symbol, None) if data is not None: algorithm.SubscriptionManager.RemoveConsolidator(removed.Symbol, data.Consolidator) for security in changes.AddedSecurities: symbol = security.Symbol newsAsset = algorithm.AddData(TiingoNews, symbol) self.newsData[symbol] = NewsData(newsAsset.Symbol) for security in changes.RemovedSecurities: newsData = self.newsData.pop(security.Symbol, None) if newsData is not None: algorithm.RemoveSecurity(newsData.Symbol) class SymbolData2(object): def __init__(self, algorithm, security, fastPeriod, slowPeriod, signalPeriod, movingAverageType, resolution): self.Security = security self.MACD = MovingAverageConvergenceDivergence(fastPeriod, slowPeriod, signalPeriod, movingAverageType) self.Reg = RegressionChannel(50, 2) self.Consolidator = algorithm.ResolveConsolidator(security.Symbol, resolution) algorithm.RegisterIndicator(security.Symbol, self.MACD, self.Consolidator) algorithm.RegisterIndicator(security.Symbol, self.Reg, self.Consolidator) self.PreviousDirection = None def WarmUpIndicators2(self, history): for index, row in history.loc[str(self.Security.Symbol)].iterrows(): self.MACD.Update(index, row["close"]) self.Reg.Update(index, row["close"]) class MaximumDrawdownPercentPerSecurityCustom(RiskManagementModel): def __init__(self, maximumDrawdownPercent = 0.001, reenter_after_days = 1): self.maximumDrawdownPercent = -abs(maximumDrawdownPercent) self.maximumProfitPercent = abs(0.002) self.liquidated_dates_by_symbol = {} self.reenter_after_days = timedelta(reenter_after_days) def ManageRisk(self, algorithm, targets): symbols_to_pop = [] for symbol, liquidation_date in self.liquidated_dates_by_symbol.items(): if algorithm.Time - liquidation_date >= self.reenter_after_days: symbols_to_pop.append(symbol) for symbol in symbols_to_pop: self.liquidated_dates_by_symbol.pop(symbol, None) targets = [] for kvp in algorithm.Securities: security = kvp.Value pnl = security.Holdings.UnrealizedProfitPercent if pnl > self.maximumProfitPercent or pnl < self.maximumDrawdownPercent or security.Symbol in self.liquidated_dates_by_symbol: # liquidate targets.append(PortfolioTarget(security.Symbol, 0)) if algorithm.Securities[security.Symbol].Invested: self.liquidated_dates_by_symbol[security.Symbol] = algorithm.Time return targets class NewsData(): def __init__(self, symbol): self.Symbol = symbol self.Window = RollingWindow[float](100)Derek Melchin
Hi Michael,
The thing slowing down the algorithm the most is the fact that it makes a History call for each security in the universe when a new data point is passed to the Update method. To resolve this, we can move the indicator warm-up logic to inside the SymbolData2 constructor.
class SymbolData2(object): def __init__(...): ... history = algorithm.History(security.Symbol, 100, Resolution.Daily) self.WarmUpIndicators2(history)With this change, we just need to add
if not (sd.MACD.IsReady and sd.Reg.IsReady): continueto the Update method after the algorithm analyzes the TiingoNews data.
Secondly, the backtesting speed can be increase by removing most of the imports from the top of the file. Without explicitly importing the python modules, LEAN will utilize the C# equivalents.
Finally, we can speed up the OnSecuritiesChanged method by only looping through `changes.AddedSecurities` and `changes.RemovedSecurities` once.
See the attached backtest which implements the suggestions above.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Fireball
Wow it's way faster! Thanks for the fundamental knowledge!
Jared Broad
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!