UPDATE 2022-01-19: We have received word from FERC that access to the historical data discussed below will be restored this week. As it becomes available we will also archive it on Zenodo just in case. Thank you to everyone who reached out and helped bring this issue to FERC’s attention!
This week we discovered that decades worth of energy system data collected by the Federal Energy Regulatory Commission (FERC) had been removed from the agency’s website. They apparently have no plan to archive it or migrate it to another platform. We are attempting to obtain a bulk download of all this data so we can archive it alongside our other raw data sources on Zenodo.
This data records many financial, operational, and economic aspects of the US energy system. It is a unique and valuable resource for anyone trying to understand how public policy and market conditions have shaped our energy system over time. Simply deleting this data with no warning, no plan to archive it, or migrate it to another platform is completely unacceptable.
If you know someone within FERC who can help get us a copy of this data to archive publicly, please put us in touch: hello@catalyst.coop
From FoxPro to XBRL
After 30 years, FERC is finally moving on from the obsolete Visual FoxPro database format. Starting this year several datasets that have historically been published using FoxPro will instead use the eXtensible Business Reporting Language (XBRL). XBRL is a dialect of XML that’s used by the US Securities and Exchange Commission and some European financial regulators to publish financial reporting.
Due to this migration, FERC is (understandably) dropping support for their obsolete Form Viewer application. The raw data used by the FERC Form Viewer has long been available for download from an agency FTP server in bulk, and with some work could be parsed and translated into modern database formats using open source libraries like dbfread.
FERC forms that used this data format and are now being migrated to XBRL include Form 1, Form 2, Form 6, Form 60, and Form 714. FERC has already migrated the last 10 years worth of data from these forms into XBRL (available for download here), but in some cases the historical data stretches back 30 years! What will happen to the older data? When we learned of the migration to XBRL, we assumed that the older raw data would remain available for download and independent use, even if the application it had originally been designed for was no longer supported. Unfortunately it seems we were mistaken.
Decades of Deleted Data
Most of this data was being distributed via an FTP server (ftp://eforms1.ferc.gov) which now seems to have been taken offline, though the FERC 714 data is still downloadable from the website. In addition, other data that is not being migrated to XBRL was also hosted on this FTP server, including older FERC Electric Quarterly Reports (EQR) covering 2002-2013. That data no longer appears to be available anywhere online. There may have been other data on the FTP server.
Data which no longer appears to be available from FERC in any format as a result of this FTP server going down include:
- Electricity: FERC Form 1/1F/3Q for 1994-2010 (17 years)
- Gas: FERC Form 2/2A/3Q for 1991-2010 (20 years)
- Oil: FERC Form 6/6Q for 2000-2010 (11 years)
- Centralized Service Companies: FERC Form 60 all years up to 2010, but they aren’t enumerated elsewhere.
- Electricity Market Transactions: FERC EQR for 2002-2013 (12 years)
We had already archived all years of FERC Form 1 on Zenodo as part of the PUDL project, and were planning to do the same for the Form 2 and EQR later this year.
What Happens Now?
When we discovered that the old data was no longer available on the morning of Thursday, 2022-01-13, we emailed the ferconlinesupport@ferc.gov account to find out what was going on, they responded that the Visual FoxPro software was no longer supported, and directed us to the (incomplete) migrated XBRL data. We requested that either the historical data be made available elsewhere on the FERC website, or that we be given the opportunity to download the data in bulk so we could archive it ourselves. As of the end of day Friday, 2022-01-14 we have received no response to this request, and are worried that the data may no longer be available in any form.
We would like to get in touch directly with someone inside FERC that might have access to a backup of the FTP server or other internal archives so that we can rescue this data and ensure that it remains available to the the public. If you have a contact within FERC that might be able to help, please put us in touch!
2 replies on “Rescuing Historical FERC Data”
My name is Shane Vyskocil, and I am currently embarking on research into the oil pipeline industry and have found much of the data I would like to use within FERC Form No. 6. I see the work you have done incorporating FERC Form 1 into the PUDL. Do you have a similar data data set with Form No. 6 data compiled into a single data set, by company and year basis?
I’m so sorry it took this long to approve your comment!
We haven’t had a reason to dig into the Form 6 yet, but we’ve started archiving it (both it’s old DBF and new XBRL formats) so that it’s versioned, and programmatically available at least. You can find the raw data archive on Zenodo.