I am trying to do some of my own programmatic analysis of data recently purchased and downloaded from QC, specifically it is GBPUSD in second resolution. When trying to read these files programmatically in C#, I get an error, “System.IO.InvalidDataException: 'A local file header is corrupt.'” This also happened to me with GBPUSD hourly data, which was not too big of an issue thanks to the workaround below.
The way I got around this error is by manually extracting the .zip, then zipping the files back into a .zip. When trying to access this rezipped file programmatically, it works fine. Unless someone can correct me, this suggests that something is wrong with the original downloaded .zip files. I want to ask here what's wrong before I resort to scripting to massage these zip files. Since this is paid for data, I hope that there is no corruption going on. Am I doing something wrong to access these files or is there something wrong with how these files are zipped?
List<BarData> barDatas = new List<BarData>();
using (ZipArchive archive = ZipFile.OpenRead(barDataFilePath))
{
foreach (ZipArchiveEntry entry in archive.Entries)
{
if (entry.FullName.EndsWith(".csv", StringComparison.OrdinalIgnoreCase))
{
using (StreamReader streamReader = new StreamReader(entry.Open()))//Error occurs here
{
}
}
}
}
Jared Broad
Hey Midaroh!
Please drop a support ticket with the specific dates/tickers, and we can look into the cloud files to ensure they're OK. I know sometimes there are operating system sensitivities with the zip readers, and maybe those files are not happy on Windows/Linux etc.
Either way if we'll replace the files if needed for free/grant you credit,
Best
Jared
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
Midaroh
Thanks Jared. Though it looks like I need a monthly subscription to a support seat in order to report the issue. Is there some other way to report these things?
Midaroh
As a note, I am using Windows 10 and the latest .NET 6.
Also for anyone else who comes across this, a workaround for fixing many datafiles at a time is to use a tool to repair the .zip archive. These are my steps:
Back up or copy your datafiles out of their folder to be safe
Using WinRAR:
Navigate to the folder containing the .zip data files
Select all files
Tools > Repair archive > Check treat as ZIP > OK. You will now have rebuilt.*.zip archives of all the data files.
Delete/Move the original files out of the folder.
Fix the names of the repaired archives using Powershell:
get-childitem *.zip | foreach { rename-item $_ $_.Name.Replace("rebuilt.", "") }
Now I can read the files in C# using StreamReader.
Midaroh
The material on this website is provided for informational purposes only and does not constitute an offer to sell, a solicitation to buy, or a recommendation or endorsement for any security or strategy, nor does it constitute an offer to provide investment advisory services by QuantConnect. In addition, the material offers no opinion with respect to the suitability of any security or specific investment. QuantConnect makes no guarantees as to the accuracy or completeness of the views expressed in the website. The views are subject to change, and may have become unreliable for various reasons, including changes in market conditions or economic circumstances. All investments involve risk, including loss of principal. You should consult with an investment professional before making any investment decisions.
To unlock posting to the community forums please complete at least 30% of Boot Camp.
You can continue your Boot Camp training progress from the terminal. We hope to see you in the community soon!