Share New Research 
Hi,
I'm new to QC platform.
I want to incorporate my ML model saved as pkl file in backtesting and later live trading on quantconnect.
My first ptoblem is that I don't know how to calculate ta_lib indicators. I create simple example that does not work:
import pandas as pd
import numpy as np
from talib.abstract import (
DEMA, EMA, MIDPRICE, SMA, T3, TEMA, TRIMA, WMA,
ADX, ADXR, AROONOSC, BOP, CMO, DX, MFI, MINUS_DM, MOM, ROC, RSI,
TRIX , WILLR, ATR, NATR, BBANDS, AROON, STOCHRSI,
HT_TRENDLINE, AD, OBV, HT_DCPERIOD, HT_DCPHASE, HT_TRENDMODE,
TRANGE, AVGPRICE, MEDPRICE, TYPPRICE, WCLPRICE, ULTOSC,
MAMA, SAR, SAREXT, APO, MACD, ADOSC,
HT_PHASOR, HT_SINE, STOCHF, STOCH
)
# GLOBALS
periods = [5, 30, 60, 300, 480, 2400, 12000, 96000]
class CalibratedResistanceAtmosphericScrubbers(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2019, 12, 31) # Set Start Date
self.SetCash(100000) # Set Strategy Cash
self.AddEquity("SPY", Resolution.Minute)
model = self.Download("https://github.com/MislavSag/trademl/blob/master/trademl/modeling/rf_model.pkl")
def OnData(self, data):
'''OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
Arguments:
data: Slice object keyed by symbol containing the stock data
'''
open_ = self.Securities["SPY"].Open
high_ = self.Securities["SPY"].High
low_ = self.Securities["SPY"].Low
close_ = self.Securities["SPY"].Close
volume_ = self.Securities["SPY"].Volume
dema = DEMA(np.array(close_), 30)
self.Debug(f'DEMA value is equal to: {dema}')
If I ran this, it returns an error:
Runtime Error: TypeError : Argument 'real' has incorrect type (expected numpy.ndarray, got NoneType) at OnData in main.py:line 81 TypeError : Argument 'real' has incorrect type (expected numpy.ndarray, got NoneType)
Derek Melchin
Hi Mislav,
I've attached a backtest which shows how we can implement an algorithm that utilizes talib. Instead of constructing a DEMA object with an array containing 1 element, we need to ensure we pass DEMA an array that is atleast twice in length of the timeperiods parameter (30). We can accomplish this by saving a rolling window of closing prices and pass this rolling window to DEMA whenever we want the indicator calculated.
Since we are needing a rolling window, we should also make sure we warm up the rolling window before the algorithm starts trading. As our documentation shows, we can accomplish this with
self.SetWarmUp(dema_period * 2)in this situation.
See that attached backtest for the full solution file.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mislav Sagovac
Derek Melchin ,
Thank you, thats very helpfull.
Why is warm up period two times longer than lookback, shouldn't it be the same as lookback? We wait wor 3 periods and than we can strat calculating indicator. It's not so importnant in my case because I have one minute resolution (so lots of data will go in the backtest), I am jsut curious.
Now, if you can help me with additional problem it would solve most of my problems!. I have written a function that automaticly calculates lots of features (cca 315 of them) based on pandas data frame (I think it can be a numpy too.). here are the functions (you don't have to inspect this, please cont read above);
def add_ind(ohlcv, f, n, periods): """ Add technical indicator to pd.DataFrame Parameters ---------- f : function function from ta_lib package. n : str Nme prefix. Returns ------- pd.Data.Frame. """ ind = pd.concat([f(ohlcv, p).rename(n + str(p)) for p in periods], axis=1) return ind def add_ind_df(ohlcv, f, n, periods): """ Add technical indicator to pd.DataFrame when indicator has multiplie outputs. Parameters ---------- f : function function from ta_lib package. n : str Nme prefix. Returns ------- pd.Data.Frame. """ ind = [f(ohlcv, p).add_prefix((f._Function__namestr + '_' + str(p) + '_')) for p in periods] # ind = [f(ohlcv, p). # set_axis((f._Function__namestr + '_' + # pd.Series(f.output_names) + '_' + str(p)), axis=1) # for p in periods] ind = pd.concat(ind, axis=1) return ind def add_technical_indicators(data, periods): """Add tecnical indicators as featues. Arguments: data {pd.DataFrame} -- Pandas data frame with OHLC data periods {list} -- List that contain periods as arguments. Returns: pd.dataFrame -- Pandas data frame with additional indicators """ # add technical indicators for variuos periods when ind has 1 output indsList = [DEMA, EMA, MIDPRICE, SMA, T3, # MIDPOINT TEMA, TRIMA, WMA, # KAMA memory intensive! ADX, ADXR, AROONOSC, BOP, CMO, DX, MFI, MINUS_DM, MOM, ROC, RSI, TRIX , WILLR, # CCI NE RADI (VALJDA) ATR, NATR] inds = [add_ind(data, f, f._Function__name.decode('ascii'), periods) for f in indsList] inds = pd.concat(inds, axis=1) data = pd.concat([data, inds], axis=1) # add technical indicators for variuos periods when ind has multiplie # outputs indsList = [BBANDS, AROON, STOCHRSI] inds = [add_ind_df(data, f, f._Function__name.decode('ascii'), periods) for f in indsList] inds = pd.concat(inds, axis=1) data = pd.concat([data, inds], axis=1) # add tecnical indicators with no arguments indsList = [HT_TRENDLINE, AD, OBV, HT_DCPERIOD, HT_DCPHASE, HT_TRENDMODE, TRANGE, AVGPRICE, MEDPRICE, TYPPRICE, WCLPRICE, ULTOSC] inds = [f(data).rename(f._Function__name.decode('ascii')) for f in indsList] inds = pd.concat(inds, axis=1) data = pd.concat([data, inds], axis=1) # add other indicators data[['MAMA', 'FAMA']] = MAMA(data) # MAVP ne radi data[['MAMA_25', 'FAMA_25']] = MAMA(data, fastlimit=0.25, slowlimit=0.02) # MAVP ne radi data[['MAMA_5', 'FAMA_5']] = MAMA(data, fastlimit=0.5, slowlimit=0.05) # MAVP ne radi data['SAR'] = SAR(data) data['SAR_1'] = SAR(data, acceleration=0.01, maximum=0.01) data['SAR_2'] = SAR(data, acceleration=0.02, maximum=0.02) data['SAREXT'] = SAREXT(data) startvalue, offsetonreverse, accelerationinitlong, accelerationlong,\ accelerationmaxlong, accelerationinitshort, accelerationshort,\ accelerationmaxshort = np.random.uniform(low=0.01, high=0.4, size=8) data['SAREXT_rand'] = SAREXT(data, startvalue=startvalue, offsetonreverse=offsetonreverse, accelerationinitlong=accelerationinitlong, accelerationlong=accelerationlong, accelerationmaxlong=accelerationmaxlong, accelerationinitshort=accelerationinitshort, accelerationshort=accelerationshort, accelerationmaxshort=accelerationmaxshort) data['APO'] = APO(data) data['APO_1'] = APO(data, fastperiod=24, slowperiod=52, matype=0) data['APO_2'] = APO(data, fastperiod=50, slowperiod=100, matype=0) data['APO_3'] = APO(data, fastperiod=100, slowperiod=200, matype=0) data['APO_4'] = APO(data, fastperiod=200, slowperiod=400, matype=0) data['APO_5'] = APO(data, fastperiod=12000, slowperiod=24000, matype=0) data['ADOSC'] = ADOSC(data) data[['MACD', 'MACDSIGNAL', 'MACDHIST']] = MACD (data) # data[['MACD_24', 'MACDSIGNAL_24', 'MACDHIST_24']] = MACD (data, # fastperiod=24, # slowperiod=52, # signalperiod=18) # data[['MACD_48', 'MACDSIGNAL_48', 'MACDHIST_48']] = MACD (data, # fastperiod=48, # slowperiod=104, # signalperiod=36) # data[['MACD_200', 'MACDSIGNAL_200', 'MACDHIST_200']] = MACD (data, # fastperiod=200, # slowperiod=300, # signalperiod=50) # data[['MACDEXT', 'MACDEXTSIGNAL', 'MACDEXTHIST']] = MACD (data) # data[['MACDEXT_', 'MACDEXTSIGNAL_', 'MACDEXTHIST_']] = MACD (data, # fastperiod=200, fastmatype=0, # slowperiod=100, # slowmatype=0, # signalperiod=50, # signalmatype=0) # data[['MACDFIX', 'MACDFIX SIGNAL', 'MACDFIXHIST']] = MACDFIX(data) # data[['MACDFIX_18', 'MACDFIX SIGNAL_18', # 'MACDFIXHIST_18']] = MACDFIX(data, 18) # data[['MACDFIX_50', 'MACDFIX SIGNAL_50', # 'MACDFIXHIST_50']] = MACDFIX(data, 50) # data[['MACDFIX_200', 'MACDFIX SIGNAL_200', # 'MACDFIXHIST_200']] = MACDFIX(data, 200) # data[['MACDFIX_12000', 'MACDFIX SIGNAL_12000', # 'MACDFIXHIST_12000']] = MACDFIX(data, 12000) data[['inphase', 'quadrature']] = HT_PHASOR(data) data[['sine', 'leadsine']] = HT_SINE(data) data[['fastk', 'fastd']]= STOCHF(data) data[['fastk_20', 'fastd_20']]= STOCHF(data, fastk_period=20, fastd_period=9, fastd_matype=0) data[['fastk_200', 'fastd_200']]= STOCHF(data, fastk_period=200, fastd_period=80, fastd_matype=0) data[['fastk_3600', 'fastd_3600']]= STOCHF(data, fastk_period=3600, fastd_period=400, fastd_matype=0) data[['slowk', 'slowd ']]= STOCH(data) data[['slowk_30', 'slowd_30']]= STOCH(data, fastk_period=30, slowk_period=15, slowk_matype=0, slowd_period=9, slowd_matype=0) return dataNow, when I want to calculate the functions, I just execute:
periods = [5, 30, 60, 300, 480, 2400, 12000, 96000] data = add_technical_indicators(data, periods)where data is pandas dataframe with OHLCV columns. Now, here is my main question: is it possible to use append pandas df, similiar as you did with numpy in your example, to calculate all this indicators in every step, using above function? Or I have to rewrite everything and doing it indicator by indivator?
If you can help me with that most of my problems would be solved. Than I would just need to apply my ml model on those features and get prdictions.
Derek Melchin
Hi Mislav,
In order to receive at least 1 non-nan data point, the DEMA indicator requires the numpy array that is passed to it (the first argument) to be at least twice as long as the `timeperiod` (the second argument). This is why we required the `self.closes` array to be twice the length of the DEMA timeperiod in the previous algorithm I shared.
We can definitely replace the numpy array we used earlier with a pandas DataFrame. See the backtest I attached for a working example. After we update the DataFrame with the latest closing price, we pass the list of closing prices to each talib indicator method. The results from calling each indicator method are then added as a column in our DataFrame. This produces a time series of indicator values. To retrieve the latest value for each indicator, we just need to slice the bottom row with
self.rolling_window.iloc[-1]As time is a constraint, I was not able to rework and debug the `add_ind`, `add_ind_df`, and `add_technical_indicators` methods that were published. However, the backtest I attached demonstrates the principle of storing multiple indicator time series values in a single DataFrame. For a more involved 1-1 support session, I recommend checking out our new Tutoring Initiative.
Best,
Derek Melchin
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Mislav Sagovac
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!