Split/Dividend Factor Files.

I worked through the calculations on a spreadsheet.  The results aren’t exact for the dividend adjustment, but probably close enough.
Looks like the Factor file columns are [Date of Previous Close, Dividend Adjustment, Split Adjustment].
I believe the calculations are as follows
Dividend Adjustment  = Prev Adj - Dividend / ( Prev Close * Split Adj)
Split Adjustment  =((Prev Close / Split ) / Prev Close)

Am I missing anything? 
 

Factor files have three colums: date, price factor (e.g.: dividend) and split factor.
The last value is always unity, because if we multiply the stock price for it, it will not change (be adjusted).

For instance, AAPL had three splits:

Prices are adjuted for split on the last trading day before the event and previous splits are "refactored":

Dividends factors are a little harder to reverse engineer; but from the LEAN Project the Dividend.cs file, and the FactorFile.cs

You take the current factor / previous factor to get the factor ration and then apply that to the closes to get the price.

We absolutely need this in LEAN :) If you can make a toolbox Yahoo Split and Factor file generator it would be a huge help :)

>We absolutely need this in LEAN :) If you can make a toolbox Yahoo Split and Factor file generator it would be a huge help :)

I agree.  Being able to write algos in a real IDE with a debugger puts Lean at a huge advantage over the competition.  The first step was open sourcing the engine, which was done. The next hurdle remains getting local data. Once that hurdle is satisfied I suspect an increased rate of contributions from novices.  As these people aren’t typically going to drop thousands of dollars on purchasing data just to be able to play around.

Back to the topic.

My approach will be to assume daily data was already downloaded. In that case I’ll need to go lookup the previous close price. I’m unfamiliar with the Lean framework, so what’s the optimal way to retrieve this daily data from within our new split/div downloader?  I see LeanDataWriter.LoadHourlyOrDailyFile is kinda what I need, but I’m guessing there is a cleaner way that will return a nice data structure rather than dealing with file parsing.

> Id assume it was independent from local data and use Yahoo.
This is what I ended up doing.  However, I see a potential problem if someone is using the incremental merge capability of the Yahoo downloader. For example. Downloading a single day at the end of the trading day.  The easiest way around this for now is just to download the entire stock history every time.  Maybe a slightly more efficient way that keeps the merge capability is to look for any new split/dividend events within in window that’s specified. If any exists then force a full history download, otherwise do a merge.

Here are my generated results for AAPL. I can't figure out why the values are off slightly when compared to the values in git. The first dividend calculation on 20151104 results in 0.9957377 vs. 0.9957381

My equation looks like this
0.9957377 = PrevDividendAdjustment - (Dividend / (PeviousClose * PrevSplitAdjustment ));
0.9957377 = 1 - (0.52 / (122 * 1));

My Results
19980102,0.9288142,0.0357143
20000620,0.9288142,0.0357143
20050225,0.9288142,0.0714286
20120808,0.9288142,0.1428571
20121106,0.9330894,0.1428571
20130206,0.9376360,0.1428571
20130508,0.9434302,0.1428571
20130807,0.9500057,0.1428571
20131105,0.9565650,0.1428571
20140205,0.9623695,0.1428571
20140507,0.9683196,0.1428571
20140606,0.9738740,0.1428571
20140806,0.9738740,1
20141105,0.9788234,1
20150204,0.9831409,1
20150506,0.9870720,1
20150805,0.9912316,1
20151104,0.9957377,1
20501231,1,1

From GIT
19980102,0.9311149,0.0357143
20000620,0.9311149,0.0357143
20050225,0.9311149,0.0714286
20120808,0.9311149,0.142857
20121106,0.9351125,0.142857
20130206,0.9393839,0.142857
20130508,0.9448594,0.142857
20130807,0.9511153,0.142857
20131105,0.9573959,0.142857
20140205,0.9629841,0.142857
20140507,0.9687481,0.142857
20140606,0.9741588,0.142857
20140806,0.9741588,1
20141105,0.9790044,1
20150204,0.9832496,1
20150506,0.9871304,1
20150805,0.9912512,1
20151104,0.9957381,1
20501231,1.0000000,1
 

It looks like the closing price used to generate the value of 0.9957381 we find in the github file is $122.01:
0.9957381 = 1 - (0.52 / (122.01 * 1))

From my experiece, different platforms display different adjusted prices when we go back enough in time, so this slight discrepancy is expected. 

This reinforces the wisdom of Jared's proposed assumption, since it is better to know from where all your variables in your equiations come from and their value.

Jeff, I think your calculations are correct.  I also think Alexandre is probably right too, although I'm not sure where the data came from if that's the case.  Everywhere I have looked, including the data that comes from QuantConnect, the closing price on 11/4/2015 is $122.00.

I have created a pull request on github that expands the ability of the YahooDataDownloader to download split and dividend data. I also created a utility that can create a FactorFile object.  Using the Yahoo Split/Dividend data and the AAPL daily equity data, my utility (FactorFileGenerator) was able to confirm Jeff's calculations (up to a small rounding error).

https://github.com/QuantConnect/Lean/pull/592

Nice! I’m glad to have you take this one over. :)
I have a decent implementation working, but my code is incredibly stove piped.  
The only thing I didn’t notice in your pull request is how you are handling a split and dividend events that occur on the same day.  Dividends should be applied pre-split in most cases.

Here is my code if you would like to compare the output results.  
Program.cs : https://drive.google.com/open?id=0B7uLVigTKyqrRVB1Sm9uOW9oQmM
YahooDownloader.cs : https://drive.google.com/open?id=0B7uLVigTKyqrbmNfUXh3emhzVG8

Program.cs has a bunch a symbols hacked in for testing.  I also adding an option to cache the yahoo data locally on the first pass.  After the first run just set downloader.useLocalCopy = true  to regenerate the factor files from the cached copy.
 

Thanks Jeff! Your code will definitely be helpful.  I do have some questions about your feedback.  I think a better place to discuss these issues would be on the github.  I'm going continue the discussion there, if that's alright?

https://github.com/QuantConnect/Lean/pull/592

Hi Andrew Hart  Jeff Miller  ,

I have just discovered these thread. Is it possible to download factor files for choosen tickers using YahooDownloader or some other toolbox?

I can retrieve yahoo daily data but factor files are not downloaded by simply calling YahooDownloader.

I can't find the file in Lean/Toolbox that can download factor files.

We're creating a new subscription product to get all QC's map and factor files updated daily. It should be ready end of next week.

Yahoo's won't cut it for true universe selection. 

Jared Broad thanks on info! That's exactly the product I need right now :)

Do you know what will be the price for the subscription? Will it be included in current plans for web QC?

It's an external vendor so will not be able to be included in the plans. We'll try and make it easy for you to get LEAN setup locally but it's not cheap. 

Thanks, how can I know when the service will be ready?

I'll post here to let you know Mislav 

Hi Jared Broad ,

Have you ever made factor file available for purschase?

I am strugling with my local data set up, so maybe I would buy complete factor and map files if available.

I have "raw" data with only symbols, but I can't find publicly avaialble file which desrcibes ticker changes through time.

Best,

Mislav 

Hi Mislav;

We've made a plan for this and scheduled the work. It should be done in the next 2 weeks. The plan requires a daily update to the map and factor file as they become out of date as soon as a company releases new dividend information or splits. We're going to sell the data for a flat subscription of $600 per year for maps, factors, and $240 for coarse universe data.

The daily price data will be available at a per-day rate we're yet to decide. We redesigning the data download portal to allow you to check out almost all of the data we have available in our cloud in LEAN format. Perfectly curated and maintained by the QC team. We hope this paired with the LEAN CLI will make it incredibly easy to run backtests locally and in the cloud. 

Best,

Jared

Thanks for the feedback. It would make local development much easier for sure. This is one of the biggest botleneck for me right now. I would be glad to pay. It isvery valuable and we have to suport your great work :)