Share New Research 
Hello,
I am trying to feed the custom consolidated data from Bitfinex API to QC. I posted my original question here.
Basically, the consolidated candle data from Bitfinex API provided is not exactly what I am looking for. I was wondering if there is any way to provide a custom live data feed to QC?
In particular, I would like to feed the data using the below function when doing live trading in stead of the data from calling Bitfinex API that I provided in my sample algorithm.
import requests
import pandas as pd
from datetime import datetime, timedelta
# EVERY 5 Mins, go to bitfinex collect all trades in the past 5 minutes and aggregate the data
def get_5mins_Bitfinex():
end_time = datetime.now() # past 5 minutes time
start_5mins = end_time - timedelta(minutes=5) # convert current time to timestamp
end_tstamp = round(datetime.timestamp(end_time)*1000,0) # convert past 5 minutes to timestamp
start_5mins_tstamp = round(datetime.timestamp(start_5mins)*1000,0)
url = 'https://api-pub.bitfinex.com/v2/trades/tBTCUSD/hist?limit=5000&start={}&end={}&sort=-1'.format(start_5mins_tstamp, end_tstamp)
response = requests.request('GET', url)
data = response.json()
data_df = pd.DataFrame(data, columns = ['id','date','volume','price'])
# aggregate the data
volume = data_df.volume.abs().sum()
volume_pos = data_df.volume.apply(lambda x: x if x >0 else 0).sum() # only take trade volume resulted from a taker buy order
open_price = data_df.price.values[0]
close_price = data_df.price.values[-1]
high_price = data_df.price.max()
low_price = data_df.price.min()
agg_data = [open_price, high_price, low_price, close_price, volume, volume_pos]
return agg_data
Best,
Hiep
Xin Wei
Hi Hieppham, Thank you for the question. Since you will be using 5-minute data for live trading, you would want to use either a scheduled event or attach a consolidator to BTCUSD (although you will not really trade based on this data) and then evaluate your trading logic using your custom consolidated data every 5 minutes, such as below.
def Initialize(self):
self.SetStartDate(2018, 10, 7)
self.SetCash(1000000)
self.AddCrypto('BTCUSD', Resolution.Minute, Market.Bitfinex)
consolidator = QuoteBarConsolidator(timedelta(minutes = 5))
consolidator.DataConsolidated += self.OnDataConsolidated
self.SubscriptionManager.AddConsolidator("BTC", consolidator)
def OnDataConsolidated(self, sender, bar):
customBTCData = self.get_5mins_Bitfinex()
def get_5mins_Bitfinex(self):
end_time = datetime.now() # past 5 minutes time
start_5mins = end_time - timedelta(minutes=5) # convert current time to timestamp
end_tstamp = round(datetime.timestamp(end_time)*1000,0) # convert past 5 minutes to timestamp
start_5mins_tstamp = round(datetime.timestamp(start_5mins)*1000,0)
url = 'https://api-pub.bitfinex.com/v2/trades/tBTCUSD/hist?limit=5000&start={}&end={}&sort=-1'.format(start_5mins_tstamp, end_tstamp)
string = self.Download(url)
## Insert method for parsing string into the dataframe you defined as data_df
## Note that Download() returns a string and not a json
## 'requests' does not work here
# aggregate the data
volume = data_df.volume.abs().sum()
volume_pos = data_df.volume.apply(lambda x: x if x >0 else 0).sum() # only take trade volume resulted from a taker buy order
open_price = data_df.price.values[0]
close_price = data_df.price.values[-1]
high_price = data_df.price.max()
low_price = data_df.price.min()
agg_data = [open_price, high_price, low_price, close_price, volume, volume_pos]
## Insert trading logic and placing orders
return customBTCData
Also, I'd like to mention that you can get Bitfinex data for BTC in live paper-trading mode by directly specifying Market.Bitfinex for Crypto data and use a 5-minute QuoteBarConsolidator as shown in the initialize above without importing external custom data.
Please let me know if you have further questions.
Hieppham
Hi Xin Wei,
Thank you for your answer, I have some questions though:
1. As the custom function to get data from Bitfinex (`get_5mins_Bitfinex`) can only access to latest 5 mins data, so it's not possible to back-test the strategy, right?
2. I am a bit confused when you suggested to insert the trading logic inside the `get_5mins_Bitfinex` function. I would like to use some Quantconnect's TAs on the custom data as well. I thought the natural process is to add the Customdata, and define the TAs in `Initialize()` function. Then create some trading logic using the value of the TA in `OnDataConsolidated()` function?
3. I tried to scan through the github directory and QuantConnect document pages, but it seems that there is no complete documentation for python or I missed something. If that's the case, could you please point me to a good documentation source.
Thanks
Xin Wei
Hi Hieppham,
Thank you for the follow-up question.
1. The strategy can be backtested. The method `get_5mins_Bitfinex` should be called as part of the consolidator (or scheduled event). My apologies if the indentation or code confused you. Please see the updated code snippet below. Additionally, you may need to change `datetime.now()` to `self.Time` to get the current algorithm time rather than the actual date.
2. You are completely right. I think your confusion is related to item 1. Regarding the strategy you want to write, you may find the `Custom Period Indicators` documentation subsection handy. The code snippet below defines an example indicator in `Initialize` and uses `Update()` method to update the indicator value with custom consolidated data every 5 minutes in `OnDataConsolidated` event handler.
3. Our `documentation` has a lot of Python code snippets to show the basics of our API. And, the Python `demonstration algorithms` listed at the beginning of documentation sections are very useful resources about implementation details. Beyond these, users often find helpful in searching relevant community discussions. For the source code of underlying classes, it is only in C# at Github as our core engine LEAN is empowered by C#.
def Initialize(self): self.SetStartDate(2018, 10, 7) self.SetCash(1000000) self.AddCrypto('BTCUSD', Resolution.Minute, Market.Bitfinex) consolidator = QuoteBarConsolidator(timedelta(minutes = 5)) consolidator.DataConsolidated += self.OnDataConsolidated self.SubscriptionManager.AddConsolidator("BTC", consolidator) self.rsi = RelativeStrengthIndex(10, MovingAverageType.Simple) # define Indicator def OnDataConsolidated(self, sender, bar): agg_data = self.get_5mins_Bitfinex() # update the indicator value with the close price of new custom data every 5 minutes. # agg_data[3] is accessing the close price, but you can use any of the values from agg_data if you want to use open, low, etc. self.rsi.Update(self.Time, agg_data[3]) if self.rsi.IsReady: # get the current RSI value rsi_value = self.rsi.Current.Value ## Insert trading logic using the RSI values, place orders, etc. def get_5mins_Bitfinex(self): end_time = self.Time # past 5 minutes time start_5mins = end_time - timedelta(minutes=5) # convert current time to timestamp end_tstamp = round(datetime.timestamp(end_time)*1000,0) # convert past 5 minutes to timestamp start_5mins_tstamp = round(datetime.timestamp(start_5mins)*1000,0) url = 'https://api-pub.bitfinex.com/v2/trades/tBTCUSD/hist?limit=5000&start={}&end={}&sort=-1'.format(start_5mins_tstamp, end_tstamp) string = self.Download(url) ## Insert method for parsing string into the dataframe you defined as data_df ## Note that Download() returns a string and not a json ## 'requests' does not work here # aggregate the data volume = data_df.volume.abs().sum() volume_pos = data_df.volume.apply(lambda x: x if x >0 else 0).sum() # only take trade volume resulted from a taker buy order open_price = data_df.price.values[0] close_price = data_df.price.values[-1] high_price = data_df.price.max() low_price = data_df.price.min() agg_data = [open_price, high_price, low_price, close_price, volume, volume_pos] return agg_dataHieppham
Thank you very much Xin Wei for your explaination and patience.
It works now! My only concern is it takes too long to download the data using self.Download() method. It takes like 20 seconds to download less than 100 records when you run the attached code snippet. It would take less than a second when I use `requests` (run locally). Do you have any suggestion to speed this up?
class CustomDataBitcoinAlgorithm(QCAlgorithm): # Initialize def Initialize(self): self.SetStartDate(2019, 6, 20) self.SetCash(1000000) self.AddCrypto('BTCUSD', Resolution.Minute, Market.Bitfinex) consolidator = QuoteBarConsolidator(timedelta(minutes = 5)) consolidator.DataConsolidated += self.OnDataConsolidated self.SubscriptionManager.AddConsolidator("BTCUSD", consolidator) self.rsi = RelativeStrengthIndex(10, MovingAverageType.Simple) # define Indicator def OnDataConsolidated(self, sender, bar): agg_data = self.get_5mins_Bitfinex() self.rsi.Update(self.Time, agg_data[3]) if self.rsi.IsReady: rsi_value = self.rsi.Current.Value self.Debug('rsi_value {}'.format(rsi_value)) ## Insert trading logic using the RSI values, place orders, etc. def get_5mins_Bitfinex(self): # download data from Bitfinex API end_time = self.Time # past 5 minutes time start_time = end_time - timedelta(minutes=5) # convert current time to timestamp end_tstamp = round(datetime.timestamp(end_time)*1000,0) # convert past 5 minutes to timestamp start_tstamp = round(datetime.timestamp(start_time)*1000,0) getdata_time = datetime.now() url = 'https://api-pub.bitfinex.com/v2/trades/tBTCUSD/hist?limit=5000&start={}&end={}&sort=-1'.format(start_tstamp, end_tstamp) string = self.Download(url) self.Debug('data downloading time: {}'.format(datetime.now()-getdata_time)) data = json.loads(string) data_df = pd.DataFrame(data, columns = ['id','date','volume','price']) self.Debug('data size {} rows'.format(data_df.shape[0])) # aggregate the data volume = data_df.volume.abs().sum() volume_pos = data_df.volume.apply(lambda x: x if x >0 else 0).sum() # only take trade volume resulted from a taker buy order open_price = data_df.price.values[0] close_price = data_df.price.values[-1] high_price = data_df.price.max() low_price = data_df.price.min() agg_data = [open_price, high_price, low_price, close_price, volume, volume_pos] return agg_dataBest,
HiepPham
Xin Wei
Hi Hiep,
Glad to see that the code worked for you! Unfortunately, we can't speed it up as it is the time needed for downloading. Since the strategy only calls the Download() method every five minutes, a 20s download time shouldn't affect the algorithm in live-trading. Please feel free to share more details of your strategy so that we can discuss further.
Hieppham
Hi Xin Wei,
I tried with only 1 record and the downloading time is still 20 seconds. The same problem with another Rest API. 20 seconds is too much for live trading I am afraid. This is the biggest bottle neck for me so far. I attached the full algorithm for your reference. Do you think there is a bug in Download()?
Link Liang
Hi Hieppham,
We have not found any bugs in Download() method. The problem could be Bitfinex has put a limitation on our IP because of the number of calls of this API. Meanwhile, supplement the live tick data feed directly with custom data source is not recommended, and we don't have any plan to support this use case. Please consider using data source with official support.
Xin Wei
Hi Hiep,
Thanks a lot for your tests and follow-up.
Great news! We do find a bug in Download() method and have fixed it now so that it isn't slow anymore. I ran a test with your attached backtest removing the 1 record limitation. It takes less than 0.5 seconds to download 300+ rows.
Feel free to follow up with any other questions regarding your strategy.
Xin
Hieppham
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!