Hi!
I'm following this example algo here to import external data :
So I've simplified the original algo to just this code here:
# QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
# Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from AlgorithmImports import *
### <summary>
### Demonstration of using an external custom datasource. LEAN Engine is incredibly flexible and allows you to define your own data source.
### This includes any data source which has a TIME and VALUE. These are the *only* requirements. To demonstrate this we're loading in "Bitcoin" data.
### </summary>
### <meta name="tag" content="using data" />
### <meta name="tag" content="custom data" />
### <meta name="tag" content="crypto" />
class CustomDataBitcoinAlgorithm(QCAlgorithm):
def Initialize(self):
self.SetStartDate(2011, 9, 13)
self.SetEndDate(datetime.now().date() - timedelta(1))
self.SetCash(100000)
# Define the symbol and "type" of our generic data:
self.AddData(Bitcoin, "BTC")
def OnData(self, data):
if not data.ContainsKey("BTC"): return
close = data["BTC"].Close
# If we don't have any weather "SHARES" -- invest"
if not self.Portfolio.Invested:
# Weather used as a tradable asset, like stocks, futures etc.
# It's only OK to use SetHoldings with crypto when using custom data. When trading with built-in crypto data,
# use the cashbook. Reference https://github.com/QuantConnect/Lean/blob/master/Algorithm.Python/BasicTemplateCryptoAlgorithm.py
self.SetHoldings("BTC", 1)
self.Debug("Buying BTC 'Shares': BTC: {0}".format(close))
self.Debug("Time: {0} {1}".format(datetime.now(), close))
class Bitcoin(PythonData):
'''Custom Data Type: Bitcoin data from Quandl - http://www.quandl.com/help/api-for-bitcoin-data'''
def GetSource(self, config, date, isLiveMode):
if isLiveMode:
return SubscriptionDataSource("https://www.bitstamp.net/api/ticker/", SubscriptionTransportMedium.Rest)
#return "http://my-ftp-server.com/futures-data-" + date.ToString("Ymd") + ".zip"
# OR simply return a fixed small data file. Large files will slow down your backtest
return SubscriptionDataSource("https://www.dropbox.com/s/0m936vz86v9ez8u/forest%20example%20bitcoin%20added%201%20line.csv?dl=1", SubscriptionTransportMedium.RemoteFile)
def Reader(self, config, line, date, isLiveMode):
coin = Bitcoin()
coin.Symbol = config.Symbol
# Example Line Format:
# Date Open High Low Close Volume (BTC) Volume (Currency) Weighted Price
# 2011-09-13 5.8 6.0 5.65 5.97 58.37138238, 346.0973893944 5.929230648356
if not (line.strip() and line[0].isdigit()): return None
try:
data = line.split(',')
# If value is zero, return None
value = data[4]
if value == 0: return None
coin.Time = datetime.strptime(data[0], "%Y-%m-%d")
coin.EndTime = coin.Time + timedelta(days=1)
#coin.Value = value
#coin["Open"] = float(data[1])
#coin["High"] = float(data[2])
#coin["Low"] = float(data[3])
coin["Close"] = float(data[4])
#coin["VolumeBTC"] = float(data[5])
#coin["VolumeUSD"] = float(data[6])
#coin["WeightedPrice"] = float(data[7])
return coin
except ValueError:
# Do nothing, possible error in json decoding
return
Emiliano Fraticelli
the code above works with a simplified data file in .csv format.
this one, to be correct: https://www.dropbox.com/s/0m936vz86v9ez8u/forest%20example%20bitcoin%20added%201%20line.csv?dl=1
Now I'm trying to use this simplified version to load data stored in another format, and then from there possibly build a strategy and so on…
This below is a fragment of this other external dataset…
https://www.dropbox.com/s/qt3a9np8yttsxma/orats%20reduced%20strikes%20with%20changed%20date%20format.CSV?dl=1
Emiliano Fraticelli
I'm using the code below, slightly modified from the original, but it seems I can't read data in that datasets…
Emiliano Fraticelli
I don't want to do anything too complex at the moment…I just want to output on log the data in
https://www.dropbox.com/s/0m936vz86v9ez8u/forest%20example%20bitcoin%20added%201%20line.csv?dl=1
Foren Power
Unfortunately, I'm not good enough at programming to solve it. But just in case it would help to bounce ideas off someone.
I'm not completely clear on where the error is happening.
Previous:
Seems like you got it to work successfully with bitcoin data.
Current:
However, when testing on stock symbol “A”, then it's not working.
Suggestion:
from the code, it says
coin.Time = datetime.strptime(data[1], "%Y-%m-%d")
from the csv a column says
“tradeDate”
→change “datetime” to “tradeDate”
Emiliano Fraticelli
No Foren Power
datetime is a Python object whereas tradeDate is the name of a column….
we don't access columns via their names in the code but via their index…like data[1] or data[2]
There's some little bug somewhere…please wait a while let's see if we get some help
Foren Power
Thanks for the patience and teaching me about datetime. Still don't have a complete answer for you. Here's a partial one.
Change dl=1 to raw=1 to render a dropbox/github dataset into easily readable format for quantconnect. I learned this from none other than picklestirfry himself.
original btc follow by strikes:
https://www.dropbox.com/s/0m936vz86v9ez8u/forest%20example%20bitcoin%20added%201%20line.csv?dl=1
https://www.dropbox.com/s/qt3a9np8yttsxma/orats%20reduced%20strikes%20with%20changed%20date%20format.CSV?dl=1
updated btc follow by strikes:
https://www.dropbox.com/s/0m936vz86v9ez8u/forest%20example%20bitcoin%20added%201%20line.csv?raw=1
https://www.dropbox.com/s/qt3a9np8yttsxma/orats%20reduced%20strikes%20with%20changed%20date%20format.CSV?raw=1
Foren Power
Sorry the above one is shaking, not sure why.
Thanks for the patience and teaching me about datetime. Still don't have a complete answer for you. Here's a partial one.
Change dl=1 to raw=1 to render a dropbox/github dataset into an easily readable format for quantconnect. I learned this from none other than picklestirfry himself. My preferred format. from: https://youtu.be/L5FTg4PXbfc
https://www.dropbox.com/s/0m936vz86v9ez8u/forest%20example%20bitcoin%20added%201%20line.csv?raw=1
https://www.dropbox.com/s/qt3a9np8yttsxma/orats%20reduced%20strikes%20with%20changed%20date%20format.CSV?raw=1
And if you want it to look more beautiful: /s/raw=1 from: https://help.dropbox.com/files-folders/share/force-download
for example:
https://www.dropbox.com/s/qt3a9np8yttsxma/orats%20reduced%20strikes%20with%20changed%20date%20format.CSV?/s/raw=1
Foren Power
The date has to be in the first column.
I believe that I triangulated the error in custom data import.
Aside, I could have been wrong before about the data on dropbox before. But uploading to github makes it easy to get the raw version.
When the datetime column isn't in the data[0] or first column from left to right it doesn't work.
Tried numbers and letter in the first column and the datetime is set to data[1]. That fails.
Now that we know where the problem is. Next goal is how to fix it. No clue. Will continue working on it.
Louis Szeto
Hi Emiliano
We believe you wish to use this link's data:
https://www.dropbox.com/s/qt3a9np8yttsxma/orats%20reduced%20strikes%20with%20changed%20date%20format.CSV?dl=1
given your code snippet's attribute names, so the link of data scraping shall be changed.
From the data, the first column is the ticker's symbol value, which is not a digit. Thus, the line if not (line.strip() and line[0].isdigit()): return None will skip all your data input, instead we suggest replacing with this to skip the header row where all labels are not digited:
Also, data attributes store in a hash table should be called with .GetProperty('strike') instead of directly calling it a method. In lines 45 & 47, the variable should match its set name in line 37.
The attached backtest is an example using "CallBidPrice" (the columns with index [14]) as the data price value for ease of visualization.
Best
Louis Szeto
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Emiliano Fraticelli
Thanks a lot! I forgot about that line…sorry!
I won't accept the answer just because if I do the discussion will be closed, if I have other problems I would like to make questions here so that I don't overwhelm of posts the forums…..
If that's OK for you, obviously…!
Emiliano Fraticelli
Well, I'm already having a new problem…
So this is a fragment of .CSV closer to the actual, final one I'll be using.
As you can see dates are different than in the previous .CSV.
I've modified my code accordingly but it seems I can't read it….
I've basically changes just this line:
But it seems that the system does not like it
Emiliano Fraticelli
So it's either the download link that is not okay (I've put dl=1) or the date format…
Foren Power
So far unsuccessful on this puzzle.
Attached it as a backtest because when I try to copy the code directly, it says indent mismatch. Just incase that helps anyone look at it. Probably not necessary.
Between the csvs there is one difference. The former contains only one symbol in the “ticker” column. The ladder contains multiple symbols in that column.
Louis Szeto
Hi Emiliano, Foren
Emiliano, it should be the DateTime format. Please use a notepad to open the .csv file and read as text instead of excel, as excel would change the display of the date format.
Foren, indent mismatch would just be the tab spacing is not correctly aligned, probably mixing some "space" within your "tab". Also, multiple symbols should be separated into multiple columns.
Best
Louis Szeto
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Foren Power
Greetings Louis, Emiliano, and QC forums,
Let me see if I get what you are saying…multiple columns are required for multiple ticker symbols.
For the goal of getting orats options data to work with quant connect locally, should I parse the data to match the old algoseek data format (because I can't afford algoseek data yet, hopefully one day). Then it may work with local backtests and trading thanks to: Lean/ToolBox/AlgoSeekOptionsConverter.
What am I missing here? Let me know if you think of a more efficient and simple method to work with options data.
Emiliano Fraticelli
It works now…but don't think the problem was the one that Louis Szeto told.
The problem was that start and end dates of the backtests didn't match any of the dates in the CSV….now that they match..it works! Anyway, did what Louis told, me, too!
Louis Szeto
Hi Foren
To use custom option data, we recommend looking through our option data format via Data, and compose your own data based on that format. Then it is possible to import the data and set its attribute just like the above examples.
Best
Louis Szeto
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Emiliano Fraticelli
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!