Hi all,
I was trying to import custom EURUSD price data in CSV format from DropBox into LEAN for backtesting, but I was unable to do so successfully.
I tried using the NIFTY template that was provided in the QuantConnect tutorial, but when I changed the link to my dropbox link (containing the EURUSD data), the backtest failed. No trades were made and there was zero results.
Attached is my code and I would appreciate any help.
The dropbox link containing the EURUSD data (in CSV) is https://www.dropbox.com/s/6q49hksqtz48sy9/EURUSD1440.csv?dl=1
class MACDTrendAlgorithm(QCAlgorithm):
def Initialize(self):
'''Initialise the data and resolution required, as well as the cash and start-end dates for your algorithm. All algorithms must initialized.'''
self.SetStartDate(2008, 1, 1) #Set Start Date
self.SetEndDate(2018, 1, 1) #Set End Date
self.SetCash(10000) #Set Strategy Cash
# Find more symbols here: http://quantconnect.com/data
eurusd = self.AddData(EuroDollar, "EURUSD", Resolution.Daily).Symbol
self.UniverseSettings.Leverage = 50
# define our daily macd(12,26) with a 9 day signal
self.sma_actual = self.SMA(eurusd,2,Resolution.Daily)
self.sma = self.SMA(eurusd,100, Resolution.Daily)
self.psar = self.PSAR(eurusd, 0.04,0.04,0.4,Resolution.Daily)
self.rsi = self.RSI(eurusd,100, MovingAverageType.Simple,Resolution.Daily)
self.previous = datetime.min
def OnData(self, data):
# only once per day
if self.previous.date() == self.Time.date(): return
if data.ContainsKey("eurusd"):
self.today = CorrelationPair(self.Time)
self.today.CurrencyPrice = data["eurusd"].Close
'''OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.'''
# define a small tolerance on our checks to avoid bouncing
tolerance = 0.0025
buysignal = (self.sma_actual.Current.Value > self.sma.Current.Value and self.rsi.Current.Value>50 and self.sma_actual.Current.Value>self.psar.Current.Value)
sellsignal = (self.sma_actual.Current.Value < self.sma.Current.Value and self.rsi.Current.Value<50 and self.sma_actual.Current.Value<self.psar.Current.Value)
sellexit = (self.sma_actual.Current.Value > self.sma.Current.Value)
buyexit = (self.sma_actual.Current.Value < self.sma.Current.Value)
holdings = self.Portfolio["EURUSD"].Quantity
# Long Order
if holdings <= 0 and buysignal == True:
self.SetHoldings("EURUSD",1.0)
# Short Order
if holdings <= 0 and sellsignal == True:
self.SetHoldings("EURUSD",-1.0)
# Long Exit
if holdings < 0 and sellexit == True:
self.Liquidate("EURUSD")
#Sell Exit
if holdings > 0 and buyexit == True:
self.Liquidate("EURUSD")
self.previous = self.Time
class EuroDollar(PythonData):
'''EURUSD Custom Data Class'''
def GetSource(self, config, date, isLiveMode):
return SubscriptionDataSource("https://www.dropbox.com/s/6q49hksqtz48sy9/EURUSD1440.csv?dl=1", SubscriptionTransportMedium.RemoteFile)
def Reader(self, config, line, date, isLiveMode):
if not (line.strip() and line[0].isdigit()): return None
# New Nifty object
currency = EuroDollar()
currency.Symbol = config.Symbol
try:
# Example File Format:
# Date, Open High Low Close Volume Turnover
# 2011-09-13 7792.9 7799.9 7722.65 7748.7 116534670 6107.78
data = line.split(',')
currency.Time = datetime.strptime(data[0], "%Y-%m-%d")
currency.Value = data[4]
currency["Open"] = float(data[1])
currency["High"] = float(data[2])
currency["Low"] = float(data[3])
currency["Close"] = float(data[4])
except ValueError:
# Do nothing
return None
return currency
Daniel Chen
Hi Frost,
Thank you for your post and your effort to study the NIFTY example. We should modify the reader() method according to fit a new data frame and we'd better not only change the dropbox link. I would recommend you take advantage of the Pandas library to import custom data more easily. Please check out the attached backtest for details. Hope it helps!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Frost
Hi Daniel,
Thanks for taking the time out to help with my issue.
I noticed that the examples you gave, as well as the examples that QuantConnect provided were centered around importing other types of data into LEAN (eg. headlines, etc.). However, can we upload price data in order extend the range of our backtests?
For instance, QuantConnect has EURUSD price data up till 2004. However, is it possible to upload EURUSD price data from 1980-2004 into LEAN and run a backtest using this system.
I apologize if I missed anything you explained as I am new to Python.
Thanks.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Alethea Lin
Hi Frost,
The only way to backtest on extended historical data is by using custom data, which you are already attempting to do. You can combine the 2 data sources by instructing the algorithm to trade on the imported custom data until 2004 and trade on QC data after 2004. Please do keep in mind that Euro came to existence in 1999, so before that, it was a proxy called ECU.
Hope this helps!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Frost
Hi Alethea,
Thanks for your reply. Would it be possible if you provided a snippet of code that instructs the algorithm to use Oanda's EURUSD data for 2004 onwards and my custom data for pre-2004?
Thanks for your assistance.
Regards,
Caleb
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Jack Simonson
Hi Caleb,
Have a look at the backtest I've attached for how to do this. I've had to make some modifications to your original code, but this should do the trick!
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Erik Bengtson
Cool James. I will take look. I might migrate what I'm doing to yours baseline implementation.I was just about to create an implementation for turtle soup strategy, better to start from a good basis
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Erik Bengtson
Thanks Jay, I will take a look at that option.
James, here my version of your baseline project. I will try to merge with the changes I've done on the previous project. The code looks cleaner.
GIT GenericTree updates
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Erik Bengtson
and of course, the current BB strategy implementation there must be only for trading ranges
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Erik Bengtson
James,
I updated some code with fixes and to make it compatible with QC
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
James Smith
Thanks Erik. I'm going to try to integrate your changes as soon as I can. I can work without a pull request but you can go that route if you prefer. I'm giving genetic programming using this setup a lot of attention so feel free to suggest improvements or report any issues. The number one thing that helps me out is getting a third-party opinion on things. I have made quite a lot of changes to this and the genetic optimizer project and am getting fairly pleasing results.
In terms of an optimization rig, I have an old 4 slot server capable of 24 cores that I obtained for basically peanuts. I don't know how the costs stack up over time against cloud compute. I imagine it would even out after a few weeks of running 24/7 as long as power costs aren't too high.
For live hosting of trading algorithms +1 for digitalocean.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Erik Bengtson
James, I will check how to do a pull request. Not really familiar with that.
Next steps for me are the integration of additional signals in order of creating a few strategies. The additional signals I'm looking at is the Autochartist, integration with rest based NN services.
I will provide further feedback as soon as I progress using this framework, but immeditelly I think the configuration is rather verbose, I will change it into 2 steps, to make it more human friendly.
Thanks for this.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
James Smith
I've merged the Bollinger and Channel breakout from your fork. Seems like a great idea to allow a survival period for the approximate coincidence of signals. I'm wondering whether this could use QuantConnect.Indicators.RollingWindow?
You're right that the Optimizer configuration is unwieldy for this level of complexity. I may address that sometime soon. In the meantime I'm using a few scripts and tools to shift json around.
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!