Using Rest Custom Data returns an

Hi Mislav,

The attached backtest shows how we can use `RemoteFile` to read from the URL above.

Best,
Derek Melchin

Derek Melchin , thanks for help. I have added logs to the code. They doesn't show any data. I have also changed the start dae but should't effect the returned data.

I have 3 questions:

1) Why do we use RemoteFile instead of Rest, when we want to send GET request on every time stamp? I want to send request GET requests in both backtest and live trading.

2) Why my attached code deosn't return any data?

3) Is it possible to change source url on every time stamp? More concretly, I would like to change date query argument in evey time stamp.

Hi Mislav,

#1 - To send GET requests, we should use Rest instead. However, when using Rest with the URL provided, an access error is thrown. Generally, this means we need to add login credentials to the URL endpoint.

#2 - The data that's returned from the URL is timestamped to 2020-12-31 15:00:00, so we need to start the backtest before this date to have the data point passed to the OnData method.

#3 - Yes, this can be done by manipulating the `date` argument that's passed to the `GetSource` method.

Best,
Derek Melchin

Hi Derek,

1) I made this AI endpoint by myself using R plumber API (

). It is hosted on Digital Ocean. If it works with RemoteFile, I don't understand why it doesn't work with Rest when it is the same endpoint. I can try try to add login creds, but I am not sure if that will help.

2) I figure out that part.

3) I don't know how to do that. I want to change date on every OnData step. but I define my `GetSource` in Initialize on the beginning. How can I provide date on each step, that is change source url on every step?

UPDATE:

I have figures out how to change date in url object. I can just use date argument in `GetSource` method:

I have read on the community that this function is called every day. But I would need hour resolution, that is, I would need to call this function every hour. 

If I get it right,  SubscriptionTransportMedium.RemoteFile is meant to be used for backtesting, not live trading. And it not meant to be called multiple times in a day. In backtest, I can make it work probbably by importing data every day, but I am not sure how to than transform the code to work in live trading.

If I deploy the algo to live trading, can I make remote file to call dataevery hour?

I still don't understand why Rest doesnt work. I have aked question on digital ocean community, because th eAPI is hosted on their droplet.

Hi Mislav,

RemoteFile can be used for live trading. Refer to the source code here and this related thread. To have the GetSource method called every hour, we need to use `Resolution.Hour` when calling `AddData`.

Best,
Derek Melchin

I still can't make it work.

If I want to get the current data from my url I have to set data query parameter to last time, in my case last hour. But the problem si that date parameter in GetSource(self, config, date, isLive) always return 00:00:00 time. If I get it right RemoteFile returns data only if source (url) canges. To change my url, I need need to change date parameter, but not sure how to provide exact date (say 2021-05-12 10:00:00) and not 2021-05-12 00:00:00.

I would like to add one note. I tried the code in paper trading (with minute resolution), and on sime time stamps

Hey @Mislav; please send in a support ticket so we can investigate. 

Best, Jared

How can I use self.ttime (current time) as input to custom data.?
I have tried withdate paramter in GetSource but it always return 00:00:00 time. That is, it returns only dates and not imes.
How can I use Qc objects as arguemnta to custom data in general?

Hi Mislav,

We can access the QCAlgorithm instance from inside a custom data reader by adding a static method.

class CustomDataSource(PythonData):
   algo = None

   @staticmethod
   def set_algo(algo):
       CustomDataSource.algo = algo

However, we don't recommend doing this to gather the current time. The data stream runs ahead of the algorithm, so the algorithm time doesn't progress as the data is being read. See the attached backtest for reference.

Best,
Derek Melchin

Unfortunatelly, I can't make it work. I think I will give it up for now. I really giveit a try, I developed R code, than I made API endpoint that calculates everythng inside that R function, than I subscribe to market data because I can't send data through POST request in QC, than I wrote Custom data to subscribe to that endpoint (GET request), but it doesn't work. The code needs some time, than there is delay between current bar (say 15:00) and custom data bar (say 15:01). I don't understand why bar doesn't way for API endpoint to finish.... Rest doesn't work for some reason...

I will POST solution if I ever find one.

Derek Melchin Can you please try to put my strategy to paper trading? I an't understant why I am gettingnot found for my Custom data every time. I have traied everything but it doesn't work.

Here is the code:

from clr import AddReference
AddReference("System.Core")
AddReference("QuantConnect.Common")
AddReference("QuantConnect.Algorithm")

from System import *
from QuantConnect import *
from QuantConnect.Algorithm import QCAlgorithm
from QuantConnect.Data import SubscriptionDataSource
from QuantConnect.Python import PythonData

from datetime import datetime, timedelta
import json
import math


class ExuberAlgorithm(QCAlgorithm):

    def Initialize(self):

        # budgets for strategies
        self.budget = 100000                        # budget
        self.exuber_budget = self.budget / 2        # budget exuber
        self.exuber_agg_budget = self.budget / 2    # budget exuber aggregate
        self.confidence_threshold_exuber = 1        # thresgold for exuber strategy
        self.confidence_threshold_exuberagg = 3     # threshold for exuber agg strategy
        
        self.SetStartDate(2021, 5, 18)
        # self.SetEndDate(2020, 1, 5)
        self.SetCash(100000)
        self.symbol = self.AddData(Radf, "RADFDATA", Resolution.Minute).Symbol
        # self.symbol_radf_agg = self.AddData(RadfAgg, "RADFAGGDATA", Resolution.Minute).Symbol
        self.spy = self.AddEquity("SPY", Resolution.Minute).Symbol
        # self.spy = self.AddCrypto("BTCUSD", Resolution.Hour, Market.GDAX).Symbol
        Radf.set_algo(self)
        
        # init
        self.exuber_size = 0
        self.exuber_agg_size = 0
        self.window_close = RollingWindow[float](2)
        
    def OnData(self, data):
        
        # close rolling window
        if not data.ContainsKey(self.spy):
            self.Log(f'Not data for SPY!')
            return
        spy_close = data[self.spy].Close
        self.Log(spy_close)
        # self.window_close.Add(spy_close)
        # if not self.window_close.IsReady:
        #     return

        # Check for exuber data
        if not data.ContainsKey(self.symbol):
            self.Log(f'Not ready')
            return
        self.Log(f'Data Received. Value: {data[self.symbol].Value}, adf: {data[self.symbol].GetProperty("adf")}, date: {data[self.symbol].GetProperty("date_")}, date_QC: {self.Time}, index time: {data[self.symbol].Time}')
        
        # exuber alpha
        indicator_exuber = data[self.symbol].GetProperty("adf")
        self.Plot("Indicators", "Inidacator Exuber", indicator_exuber)
        self.Plot("Indicators", "Threshold", self.confidence_threshold_exuber)
        if self.exuber_size == 0 and indicator_exuber < self.confidence_threshold_exuber:
            self.exuber_size = math.floor(self.exuber_budget / spy_close)
            self.buy_order = self.MarketOrder(self.spy, self.exuber_size)
            self.Log(f"Buy Exuber. Exuber size after buy {self.exuber_size}")
        elif self.exuber_size > 0 and indicator_exuber > self.confidence_threshold_exuber:
            # current_return = self.window_close[0] - self.window_close[1]
            # if current_return < 0:
            self.sell_order = self.MarketOrder(self.spy, -self.exuber_size)
            self.last_pl = self.buy_order.AverageFillPrice - self.sell_order.AverageFillPrice
            self.exuber_budget = self.exuber_budget + self.last_pl
            self.exuber_size = 0
            self.Log("Sell Exuber")

        # # If radf agg data not readz cont
        # if not data.ContainsKey(self.symbol_radf_agg):
        #     self.Log(f'Radf agg not ready')
        #     return
        # self.Log(f'Data Received. Value: {data[self.symbol_radf_agg].Value}, adf: {data[self.symbol_radf_agg].GetProperty("adf")}, date: {data[self.symbol_radf_agg].GetProperty("date_")}, date_QC: {self.Time}')
        
        # # exuber alpha
        # indicator_exuber_agg = data[self.symbol_radf_agg].GetProperty("adf")
        # self.Plot("Indicators", "Inidacator Exuber Agg", indicator_exuber_agg)
        # self.Plot("Indicators", "Exuber Agg Threshold", self.confidence_threshold_exuberagg)
        # if self.exuber_agg_size == 0 and indicator_exuber_agg < self.confidence_threshold_exuberagg:
        #     self.exuber_agg_size = math.floor(self.exuber_agg_budget / spy_close)
        #     self.buy_order_agg = self.MarketOrder(self.spy, self.exuber_agg_size)
        #     self.Log("Buy ExubeeAgg")
        # elif self.exuber_agg_size > 0 and indicator_exuber_agg > self.confidence_threshold_exuberagg:
        #     self.sell_order_agg = self.MarketOrder(self.spy, -self.exuber_agg_size)
        #     self.last_pl = self.buy_order_agg.AverageFillPrice - self.sell_order_agg.AverageFillPrice
        #     self.exuber_agg_budget = self.exuber_agg_budget + self.last_pl
        #     self.exuber_agg_size = 0
        #     self.Log("Sell ExubeeAgg")

# 'server.contentio.biz/REST/sharedlinks/%7b452444f1-3175-43e5-bacf-5cd0159bfe97%7d/860c125ba23db44d82af2a439617dcec949b883b018bc332b20ac8b71f2549a5/content'
# 'http://server.contentio.biz/REST/sharedlinks/%7b452444f1-3175-43e5-bacf-5cd0159bfe97%7d/860c125ba23db44d82af2a439617dcec949b883b018bc332b20ac8b71f2549a5/content'

# 2021-05-20 17:28:00 :     415.3
# 2021-05-20 17:28:00 :     Not ready
# 2021-05-20 17:28:49 :     GetSource date 2021-05-20 13:28:48.800975 2021-05-20 00:00:00
# 2021-05-20 17:29:00 :     415.31
# 2021-05-20 17:29:00 :     Not ready
# 2021-05-20 17:29:49 :     GetSource date 2021-05-20 13:29:48.800516 2021-05-20 00:00:00
# 2021-05-20 17:30:00 :     415.405
# 2021-05-20 17:30:00 :     Not ready
# 2021-05-20 17:30:49 :     GetSource date 2021-05-20 13:30:48.802846 2021-05-20 00:00:00


class Radf(PythonData):
    algo = None
    
    @staticmethod
    def set_algo(algo):
        Radf.algo = algo
    
    def GetSource(self, config, date, isLive):
        Radf.algo.Log(f"GetSource date {Radf.algo.Time} {date}")
        date_argument = date.strftime("%Y%m%d%H%M%S")
        source = "http://207.154.227.4/alphar/radf_point?symbols=SPY&date=" + date_argument + "&window=100&price_lag=1&use_log=1&time=minute"
        return SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile)

    def Reader(self, config, line, date, isLive):
        index = Radf()
        index.Symbol = config.Symbol
        Radf.algo.Log(f"Reader date {CustomDataSource.algo.Time} {index.Time}")

        if isLive:
            try:
                line = json.loads(line)[0]
                index.Time = datetime.strptime(line['datetime'], '%Y-%m-%d %H:%M:%S')
                Radf.algo.Log(f"Reader date {CustomDataSource.algo.Time} {index.Time}")
                index.Value = 1.0  # Random value
                index.SetProperty('adf', line['adf'])
                index.SetProperty('date_', str(date))
                index.SetProperty('date_', str(date.strftime("%Y%m%d%H%M%S")))
            except ValueError:
                return None
        else:
            line = json.loads(line)[0]
            index.Time = datetime.strptime(line['datetime'], '%Y-%m-%d %H:%M:%S')
            index.Value = 1.0  # Random value
            index.SetProperty('adf', line['adf'])
            index.SetProperty('date_', str(date))
            index.SetProperty('date_', str(date.strftime("%Y%m%d%H%M%S")))

        return index

Hi Mislav,

Here is a simplified version we have tested in live mode:
backtest_b6ff1c068dc1a15a295b8f12d20ffefb.html

We just needed to set the time zone of the data source to UTC:

self.symbol = self.AddData(Radf, "RADFDATA", Resolution.Minute, TimeZones.Utc).Symbol

and set `EndTime` to `utcnow`:

index.EndTime = datetime.utcnow()

Now you will have to fix the backtest part because

index.EndTime = datetime.strptime(line['datetime'], '%Y-%m-%d %H:%M:%S')

is not in UTC.

Best,
Derek Melchin

I answered to the support via email, but I am sending the code with comments here too:

from datetime import datetime, timedelta
import json

class ExuberAlgorithm(QCAlgorithm):

    def Initialize(self):
        self.SetStartDate(2021, 5, 18)
        self.AddEquity("SPY", Resolution.Minute)
        self.symbol = self.AddData(Radf, "RADFDATA", Resolution.Minute, TimeZones.Utc).Symbol

    def OnData(self, data):
        
        self.Log(', '.join([f'{x}' for x in data.Values]))
        
        # when there are SPY data there are no RADF data
        # when there are RADF data there are no SPY data
        if not data.ContainsKey(self.symbol):
            self.Log(f'Not ready')
            return
        else:
            self.Log(f'There is custom data: {data[self.symbol].GetProperty("adf")}')
            
        # It never buys share
        if not self.Portfolio["SPY"].Invested:
            self.SetHoldings('SPY', 1)

class Radf(PythonData):
    
    def GetSource(self, config, date, isLive):
        date_argument = date.strftime("%Y%m%d%H%M%S")
        source = "http://207.154.227.4/alphar/radf_point?symbols=SPY&window=100&price_lag=1&use_log=1&time=minute&date=" + date_argument
        return SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile)


    def Reader(self, config, line, date, isLive):
        index = Radf()
        index.Symbol = config.Symbol
        line = json.loads(line)[0]

        if isLive:
            index.EndTime = datetime.utcnow()
        else:
            index.EndTime = datetime.strptime(line['datetime'], '%Y-%m-%d %H:%M:%S')
            index.SetProperty('date_', str(date.strftime("%Y%m%d%H%M%S")))

        index.Time = index.EndTime - timedelta(minutes=1)
        index.Value = line['adf']
        index.SetProperty('adf', line['adf'])
        return index