Can't make AddData work

Hi Rich,

First of all, is that realistic? Will QuantConnect synthesize one stream of data where there might be gaps?

If the SPY and DFF data have the same EndTime, LEAN will pass them to your algorithm in a single Slice.

Perhaps I've overlooked the DFF data source?

There is currently no DFF data in the FRED integration.

GetSource doesn't seem to have a way to accept the API key and observation start date.

You can hardcode your API key into the source URL. If you need to make the observation date dynamic, you can use the date argument. If you just want to store the API and observation date outside of the method, define some class variables in the PythonData definition or some module/global variables.

Where is the documentation for FileFormat.UnfoldingCollection?  I am guessing that it does what I want but perhaps that's one issue?

If the API returns a json object with multiple observations, use FileFormat.UnfoldingCollection. For more information about it, see Define Custom Types.

How can I pass in API key and start date?

Either hardcode them into the source URL or save them as class/global variables.

The Reader method is never called.  Why?

Check if the API endpoint is actually returning any data.

Most of the AddData examples appear to expect to read a line of data, like a line from a CSV file.  The FRED DFF is a json file …

For a full JSON example, see JSON Format Example.

Best,
Derek Melchin

Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.

Hi Derek,

Thanks so much for getting back with me.  I really appreciate it.

You mentioned several items, including a link to the same documentation that I've been looking at.  The issue I have with it is this:  

class MyCustomDataType(PythonData):
    def GetSource(self,
         config: SubscriptionDataConfig,
         date: datetime,
         isLive: bool) -> SubscriptionDataSource:
        return SubscriptionDataSource("https://raw.githubusercontent.com/QuantConnect/Documentation/master/Resources/datasets/custom-data/unfolding-collection-example.json", SubscriptionTransportMedium.RemoteFile, FileFormat.UnfoldingCollection)

    def Reader(self,
         config: SubscriptionDataConfig,
         line: str,
         date: datetime,
         isLive: bool) -> BaseData:

        if not (line.strip()):
            return None

        objects = []
        data = json.loads(line)

        for datum in data:
            index = Nifty()                             <------------------- This throws an error, what is a "Nifty()" ???
                                                                             I now *think* this should read index = MyCustomDataType ???
            index.Symbol = config.Symbol

            index.Time = datetime.strptime(datum["Date"], "%Y-%m-%d")
            index.EndTime = index.Time + timedelta(1)
            index["Open"] = float(datum["Open"])
            index["High"] = float(datum["High"])
            index["Low"] = float(datum["Low"])
            index["Close"] = float(datum["Close"])
            index.Value = index["Close"]
            objects.append(index)

        return BaseDataCollection(objects[-1].EndTime, config.Symbol, objects)

I don't understand:

– Why am I allowed to put a breakpoint in the Reader method but it is never hit.  Is that a bug?

– If the Reader method is erroring out, I'm not getting any notification. I even tried throwing an error purposefully to no avail.

– I did manage to finally get it working by passing in my QCAlgorithm so that I could at least repeatedly call .Log.

(Credit to a discord comment by ‘Max’ on 12/9/21)

Here's the working version:

from AlgorithmImports import *

class FredDffTest(QCAlgorithm):

    def Initialize(self):

        self.start_date = datetime.strptime(self.GetParameter('start_date'), '%Y-%m-%d')
        self.SetStartDate(self.start_date.year, self.start_date.month, self.start_date.day)

        self.SetCash(float(self.GetParameter('initial_cash')))
        self.fred_api_key = self.GetParameter('fred_api_key')

        self.dff = self.AddData(self.FredFactory(), "DailyFedFundRates", Resolution.Daily).Symbol        

    def OnData(self, data: Slice):
        self.Log(data)
        return

    def FredFactory(self):
        return type('DailyFedFundRates', (DailyFedFundRates,),  dict( FredAlgorithm = self, 
                                                                      FredSeries    ='DFF', 
                                                                      FredApiKey    = self.fred_api_key, 
                                                                      FredStartDate = self.start_date.strftime('%Y-%m-%d')))


class DailyFedFundRates(PythonData):

    def GetSource(self, config, date, isLiveMode):                
        source = f"https://api.stlouisfed.org/fred/series/observations?series_id={self.FredSeries}&api_key={self.FredApiKey}&file_type=json&observation_start={self.FredStartDate}"
        result = SubscriptionDataSource(source, SubscriptionTransportMedium.RemoteFile, FileFormat.UnfoldingCollection)        
        return result

    def Reader(self, config, line, date, isLiveMode):                
        json_response = json.loads(line)
        self.FredAlgorithm.Log(line[0:50])
        
        finalTime = datetime.strptime(json_response["observations"][-1]["date"], '%Y-%m-%d')
        endTime = finalTime + timedelta(1)
        self.FredAlgorithm.Log(f'endTime: {endTime}')

        data = list()
               
        for json_datum in json_response["observations"]:            
            datum          = DailyFedFundRates()
            self.FredAlgorithm.Log('in the loop')
            datum.Symbol   = config.Symbol
            datum.Time     = datetime.strptime( json_datum["date"], "%Y-%m-%d" )
            datum.EndTime  = datum.Time + timedelta(1)
            datum.Value    = float(json_datum["value"])
            data.append(datum)                                
        
        result = BaseDataCollection(endTime, config.Symbol, data)
        
        self.FredAlgorithm.Log(f"Reader ended for endTime {endTime}")        
        return result

Thanks so much for the assist.

Regards,

Rich

Hi Rich,

I now *think* this should read index = MyCustomDataType ???

Correct.
 

Why am I allowed to put a breakpoint in the Reader method but it is never hit.  Is that a bug?

Yes, thanks for reporting this bug. To be notified when we fix this issue, subscribe to GitHub Issue 6625
 

If the Reader method is erroring out, I'm not getting any notification. I even tried throwing an error purposefully to no avail.

We weren't able to reproduce this issue. We added 

raise Exception(line)

to line 172 of the BubbleAlgorithm and we got the following error:

Error invoking CAPE.Cape data reader. Line: 2014-12,2070.65,,,235.51,2014.958333,2.26,2070.65,,,27.33855203

Best,
Derek Melchin

Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.

Hey Rich Duzenbury, I share your pain. I don't know how many times I tried to achieve reading a custom universe from a json rest and I gave up.

I literally just tried it again with this example 

but guess what, copying/pasting the code didn't work and the demonstration algorithm at the end of the page is far from the examples, like the page says about reading json and the github demo is about csv with other file structure and so.

I will try to go over your code to see if I can adapt to my needs here.

Hi Rafael Trevisan ,

Please find the example from the documentation in the attached algorithm. It works as expected. We can verify it by adding a breakpoint in the FilterFunction method and inspecting the list variable.

I removed the typing from the following line

def FilterFunction(self, data: List[StockDataSource]) -> List[str]:

to avoid adding the StockDataSource class definition before the QCAlgorithm-inherited class or defining it in another file.

If the Community have any suggestion on how to improve the Importing Data section of the documentation, please let us know. 

Best regards,
Alex

Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.
 

Thanks for replying Alexandre Catarino. I partially got it working after a lot of tries. For me, the trick was to change the SubscriptionTransportMedium from Rest to RemoteFile, which I think IMHO is misleading a bit as my data comes from a rest api and not from a remote file.

However it took much more time than I expected and the reason was the debug not working when I had the breakpoint anywhere in the code. I also couldn't log inside the GetSource or the Reader method which helped to make things a bit more challenging. I even found some post in the forum to define a static method in the PythonData class to refer to the algo so I could log, but it didn't work.

I didn't mean to be harsh. The platform is great but sometimes it's more challenging to connect pieces together than others, especially when coding in Python isn't your expertise, like for i.e., defining a Factory to be able to share variables between the algo and the other class.

Just adding another comment, I have a case here where adding a breakpoint in the very first line of the FilterSelector function won't work at all. It seems that if the GetSource or Reader gets any errors the algo isn't throwing them out and you can't debug.

Hi Rafael,

I have a case here where adding a breakpoint in the very first line of the FilterSelector function won't work at all.

This issue will be addressed by GitHub Issue #6625.

Best,
Derek Melchin

Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.

Hi everyone, 

GitHub Issue #6625 has been fixed. We can now use the built-in debugger to debug PythonData classes. 

Best,
Derek Melchin

Want to invest in QuantConnect as we build the Linux of quant finance? Checkout our Wefunder campaign to join the revolution.