Is it possible to get historical data in DataNormalizationMode.Raw

Currently we can't but its relatively easy to do though so I've created a Github issue for it and we'll add it in the next few days.

Hi,

Thank you. Will wait for the changings.

Hi, Jared,

Please, write here, when changings would be done.

As of now, SetDataNormalizationMode(DataNormalizationMode.Adjusted) and calling History(symbol, start_date, end_date, Resolution.Daily) will adjust the history all the way to the present day, when it really should only adjust to the end_date. Is this by design?

My 2 cents... I would think the most approriate behavior would be to ony adjust to the end date. Two reasons. First, this would make the results repeatable, independant of when the function is called (consider the case of using this method inside a notebook. If the notebook was run a year from now the results could be completely different if a split had occurred.). Second, a more philosophical reason, is that the results contain 'forward looking' info if adjusted past the end date. This could interject data bias into ones code if not careful.

FYI, I haven't tested the behavior of the history method but presume Young Zhang's observation is correct.

Great points Dan Whitnable , Young Zhang - we'll add it to the issues list and fix it soon. You're right that historical adjustments to today would contain information from the future (encoded and hard to interpret) that should be eliminated.

Downside is you won't be able to compare prices between history look ups. But think most people are using history in one off cases for indicators etc.

As a work around, perhaps make it so that the History(startDate, endDate, DataNormalizationMode.Raw) returns splits and dividends as well as the raw close price so that the end user can adjust manually. I've noticed that dividends and splits are not returned by the History() function.

Extending history to support other normalization modes was addressed with this PR: 

You can now set the normalization mode and if its not set the normalization mode of the algorithm will be used.

I think there is a good answer to this question, I'm not trying to be snarky:

Why would anyone want adjusted prices ever?

After all, the point of backtesting is to simulate conditions at past dates, times so I don't understand why the self.Securities[s].Price by default is a value adjusted for a stock split that hasn't happened yet, at that point in the backtest, and what on earth people are doing to cope with this, and how it can be prevented if not desired.

SetDataNormalizationMode sounds nice but has to be applied to each security individually instead of once in Initialize to operate on all stocks at all times. I don't understand what's attractive about that approach. Are there developers who want some stock prices to be adjusted and others not? If so, why? If not, then why was that method chosen? Thanks. Good luck.

for easy development/backtesting of course. :)

that is a strategic approach that not everyone likes but helps in backtesting for beginners.

after good results everyone can add stocksplits which is the way that has been choosen to develop a strategy in QC.

Lets ask Jared Broad for an official answer.

You can of course add some ideas/issues on Github as one of the advanced users to adress some changes.

During the last year a lot has changed in QC.

feel free to be a part of some changes.

garyha you should have created an issue in the lean repository adressing the problem of setting the resolution for the universe and other settings but not DataNormalizationMode.

QuantConnect added some cool features like the new framework but since more quantopians joined it is maybe time to change the strategy from new features to development quality unfluenced by members.

Like that other guy was searching for a way to access sharpe inside the algo and so on.......

Jared Broad 

Alexandre Catarino 

> it is maybe time to change the strategy from new features to development quality

That's wise. There's a related phenomenon I think in us humans, many can thrive on optimism & expectation (dopamine) and that's enough for them. Some are more hungry and pursuing a level beyond those, it's called happiness (seratonin). Rolling along, most are reluctant to roll up their sleeves and do the tough work to get out of first gear. They're satisfied just figuring at some point they might go faster.

We all have the temptation to be complacent and are only moved from it by discomfort usually. We all like compliments, and rightly so, and yet criticizers often have very deeply rooted good intentions, otherwise they'd ignore us. I'll provide an example and then pipe down. Brace yourself to feel uncomfortable (with the goal of fantastic achievement on a new level maybe growing out of it at some point) ... I find the docs quite unhelpful. It seems like around here, just saying read the docs is sort of regarded like a universal solvent (and I've heard this from others too, privately, sharing their frustrations with me after a time trying to work with QuantConnect).

For example, on SetDataNormalizationMode, the haloed documentation I was pointed to says: "You can control this with the SetDataNormalizationMode() method. The DataNormalizationMode enum has the values Adjusted (default), Raw, SplitAdjusted, and TotalReturn."

... and doesn't go into what those mean or the difference between Adjusted and SplitAdjusted. So one is left googling (didn't help me in this case), fiddling around with the code (same result from most), trying to guess whether it is behaving the way its creator intended, maybe asking another question and figuring the answer will probably be: See the docs.

I'll try not to be waxing so philosophical, it's just that you have a good start, I do want what's best for you (moving beyond first gear), and I'm frustrated like many others, and frankly, heh, I found your comment to be a delightful glimmer of light, that you would think of saying that, so I hope you'll bear with me in this one neighborly rant about humanity.  I could go on. I'll stop. :) Have a great day.

Thank you for the suggestion Garyha; we do our best but there's always room to improve. we've been writing documentation for about 3 years -- basically half the team's energy is spent on how to make QC easier (tutorials, bootcamp, documentation, videos, support). Its a time consuming process and will never be perfect =)

I've updated the docs with a table to address the points you've mentioned above. If you have other specific issues with documentation please feel free to submit to the documentation issues list -- maybe you can even help with a pull-request =). 

hm nice text Garyha,.....

as you already had seen this project was long seen with developer eyes. Used by developers and people willing to learn programming. From Jareds side the description was always there:

split

Now people are comming from different worlds and complain a little bit. yeay

Till now github search was my best friend.

Lets see what will change.......

The doc update above by Jared looks great and even has better descriptions. An idea, if quantconnect.com / docs  could operate like wikipedia, would very many people contribute?

Hi Michael, I wonder if you are doing a search from the git command line, I could try to become accustomed to that. Using this as an example for future searches ....

On the web, not finding a search within github, I went with google, specifying

site:github.com/QuantConnect/Lean/blob

(For any who don't know, that insists that all of the results have that in the URL so it is pretty powerful)

https://www.google.com/search?q=DataNormalizationMode+SplitAdjusted+site:github.com/QuantConnect/Lean/blob

5 results and not sure how to narrow that down further, it would be nice to be able to target a page like the 'split' link above with those descriptions, which I've made more readable here

Raw    The raw price with dividends added to cash book
Adjusted    The adjusted prices with splits and dividendends factored in
SplitAdjusted    The adjusted prices with only splits factored in, dividends paid out to the cash book
TotalReturn    The split adjusted price plus dividends

Best of course would be finding them via www.quantconnect.com / docs ... with its search field, convenient in the QC interface as 'Docs' on the menu, user friendly. Or another idea, maybe a search there on DataNormalizationMode and splitadjusted could also find links to github pages. As mentioned, the amount of work in doing documentation is vast, so maybe that could be quite a bit of result for just a little effort.

Anyway cool thanks.

yep as you already proved i didnt find it with google.

the QC guys make always good documentation in the code which helped me a lot thats why the search on github.

i was certain that there is a comment for DataNormalizationMode

found 33 results on 4 sites which i clicked through and after 30 seconds i found it on site 4 result 32 :)

Since QC has two libraries with python code and i dont know how to concatenate the path variable on github (history path:"Algorithm.Python") i use my repo string to search stuff in the libraries:

"self.history" repo:QuantConnect/Lean/Algorithm.Python repo:QuantConnect/Lean/Algorithm.Framework language:Python

Serge d'Adesky 

so  i searched once for serge problem combining symboldata (class) and history together which helped me to find an example.....

Jared Broad this forum/conversation does not come up on the newest-sorting when i post something hmmm strange

forum does not like me which is ok,

too much spam postings i know :)

This is such an amazing platform and project you've got here!

Is it possible to make a history call for Raw data without having to set the option on the security object?

I have been able to access the historic data in the different modes by setting the mode on the security object. Of course to do that I need to add each security to my universe first and then set the mode.  I have found that the price adjustments may be affecting dollar volume calculations.  I have been grabbing the historic data for all the symbols in raw mode, which works.  But having to add all the equityies to my universe and setting the mode really slows down the backtest.  

It seems like a nice feature would be a keywork argument to the History function to specify the data mode for all data.  For example, currently in python we do:

history = self.History(
  [ x.Symbol for x in coarse ],
  100,
  Resolution.Daily
)

With the new feature we could access the data in raw price mode, for example like this:

history = self.History(
  [ x.Symbol for x in coarse ],
  100,
  Resolution.Daily,
  dataMode=DataNormalizationMode.Raw
)

A benefit of that is that I can check historical data in different modes without affecting the price mode of my whole algorithm.  I can't be sure of this but I think it would dramatically help performance in my case, where i need to check the data for many symbols.

Thanks

I am trying to find a way to select my universe by using raw data and then use momentum on adjusted data. Clearly because of splits (reverse or otherwise) momentum calculations can not be used on raw data. Unless of course there have been no splits for a given security.

Can access be had for a universe on a daily basis to both raw and adjusted prices? Perhaps I have not searched the forum thoroughly enough yet.

Perhaps that is just what the thread started out be saying - I must use the history function and it will give me what I need.