Tag: open data

Categories
updates

FERC’s Electric Quarterly Reports (EQR) in PUDL!

With support from GridLab we were able to integrate an initial draft of the FERC Electric Quarterly Reports (EQR) into PUDL! EQR has been at the top of our data wishlist for almost half a decade and we’re excited to publish this first draft for beta testing.

What’s in EQR?

We’ve heard from PUDL users that efforts to design cost-effective decarbonization pathways are hindered by limited visibility into the financial flows and market structures that shape the electricity sector. EQR offers an enormous amount of data describing transaction-level data on prices, quantities, and contractual terms between power producers, utilities and financial intermediaries. Yet the dataset remains largely underutilized due to accessibility and quality barriers. 

In its raw form, it is published by FERC as hundreds of gigabytes of nested zip files with little documentation, so querying and analyzing EQR data requires both domain knowledge and some data engineering skills. We hope that our minimally transformed version of EQR in PUDL will reduce barriers to access and get the data into the hands of the people that need it.

The EQR data records U.S. electricity wholesale market activity in great detail–sellers report transactions and their contractual terms on timescales ranging from 5-minutes in real-time spot markets, up to multi-year power purchase agreements (PPAs). With thousands of market participants and dozens of electricity products, the EQR contains billions of records.

EQR data is organized into three main tables: identifying sellers, describing contracts, and enumerating transactions. The seller identity table provides a unique FERC company ID that is used across multiple FERC Forms and associates that ID with the company name. The contracts table records 30 attributes of contracts that were active during the quarter, and whether or not any transactions took place under them. The transactions table contains 26 fields including the unit price, quantity and electricity product being sold. 

Data Access

Tutorial Video and Python Notebook

Here’s a tutorial video showing how to access data and what you can expect to see in the data. In the video, we walk through this Python based interactive notebook which demonstrates how to access and work with the EQR data, including a toy analysis of the contracts and transactions tables. 

Documentation

Review the FERC EQR data source documentation page and the data dictionary page for the 4 EQR tables: 

PUDL Data Viewer

You can browse the data through our PUDL Data Viewer UI and download small subsets of the data.

Parquet Files from Cloud Storage

We publish Apache Parquet files of all EQR data released between 2013 and present to publicly accessible cloud storage. The files are partitioned by quarter. These will be automatically updated quarterly by the 15th of Feburary, May, August and November. You can find the data in S3 under this path:

s3://pudl.catalyst.coop/ferceqr

Each table has its own subdirectory, containing quarterly parquet files:

  • s3://pudl.catalyst.coop/ferceqr/core_ferceqr__contracts/
  • s3://pudl.catalyst.coop/ferceqr/core_ferceqr__quarterly_identity/
  • s3://pudl.catalyst.coop/ferceqr/core_ferceqr__quarterly_index_pub/
  • s3://pudl.catalyst.coop/ferceqr/core_ferceqr__transactions/

See here for more information on accessing PUDL data from cloud storage.

Feedback

There’s a bunch of additional data cleaning work that we’d like to take on if we can secure additional funding for the project. This includes tasks like cleaning and standardization of categorical data, finding and fixing invalid dates, and dealing with duplicate values.

We’d love to get some feedback from real users, hear about your use case, and figure out what issues you may have run into. You can reach out to us at [email protected], or if you run into any problems, please feel free to create an issue on GitHub. You can find the issues we’re already tracking here:

Categories
updates

2025 PUDL Annual Report

Read the 2025 PUDL Annual Report to learn about PUDL developments as well as PUDL Sustainer plans for 2026.

2025 was the first year of the PUDL Sustainers Program. We designed the program to ensure the ongoing impact and reliability of PUDL:  data and software are kept up to date, bugs are fixed, and everything gets tested and released on a regular schedule. At the same time,  we sought to foster closer relationships with the organizations that rely heavily on PUDL data for their work, giving them an opportunity to see under the hood and weigh in on important decisions.

Thanks to support from our inaugural sustainers, RMI and GridLab, we were able to launch the program and keep PUDL  up and running!

24,000 unique IPs 
Accessing PUDL data from cloud storage buckets		577 users 
Registered for the PUDL Data Viewer		70+ schools 
Accessing PUDL data
20 archives
Of new data added to Zenodo		14 citations 
In academic journals, pre-prints, and theses (including Applied Energy, The Electricity Journal, and Environmental Research: Energy)		14 new tables 
Including imputed FERC 714 and EIA 930 data, SEC 10-K outputs tracking parent-subsidiary company relationships, and AEO projected energy use data
10 conferences
Attendance and presentation at USAEE, INFORMS, EPRI, TWEEDS and others!		8 terabytes
Of PUDL data distributed through S3 and Zenodo		4 interviews 
In the MIT Technology Review, the New York Times, Salon and Les Echos 
PUDL Annual Report – 2025Download
Categories
updates

Capturing the elusive FERC EQR

The Federal Energy Regulatory Commission (FERC) collects a lot of data in its role as the regulator of US electricity markets. One dataset that we’ve had our eyes on for years, but never had the resources to integrate into PUDL (until now!) is the Electric Quarterly Report or EQR.

The EQR is the reporting mechanism FERC uses for public utilities to fulfill their responsibility under section 205(c) of the Federal Power Act (FPA) to have their rates and charges on file in a convenient form and place.

Under this section of the FPA:

Where two or more public utilities are parties to the same rate schedule or tariff, each public utility transmitting or selling electric energy subject to the jurisdiction of this Commission shall post and file such rate schedule

In modern US electricity markets, there’s more than a million of these transactions every day. This makes posting them in a “convenient form and place” non-trivial. FERC provides an online EQR report viewer where you can scroll through a list of thousands of sellers and download their filing for a particular quarter as a summary PDF. But this is no way to do analysis.

They also offer bulk downloads of the raw data as quarterly zipfiles containing CSVs or XML documents, but with more than 4 billion transactions recorded since the modern reporting era started in 2013, these formats are also not very ergonomic for bulk analysis. And even as compressed zipfiles, the full FERC EQR dataset is 100 GB. It’s far too big analyze in bulk locally with tools like pandas, let alone spreadsheets. However, with the development of tools like DuckDB and Polars over the last few years, working with this scale of tabular data has become much easier — even on a laptop!

The data locked up in the FERC EQR is potentially valuable for understanding how renewable energy power purchase agreements and battery storage prices have evolved over time, and what’s really driving rising energy costs. The EQR data is particularly valuable for less organized markets like the US Southeast and Intermountain West, where there is no ISO/RTO market data. We’re hopeful that this data can also make it easier to programmatically identify anti-competitive utility affiliate transactions and uneconomic self-dispatch by regulated monopolies.

Democratizing FERC EQR Access

With additional support from Gigawatt-tier PUDL Sustainer GridLab, Catalyst is going to create an open access, cloud-native version of the FERC EQR dataset, designed for bulk analysis.

  • Timeframe: We expect to be working on the project from now through the first quarter of 2026.
  • Data Coverage: We are initially targeting the 2013Q3 to present data, since it is all in a very similar format, while older EQR data was submitted under a different reporting regime, in a different format, and is also not as rich in detail (though we have already archived the raw 2002-2013 data).
  • Outputs: We’re planning to mirror the structure of the current data, with four main tables: transactions, contracts, filer identities, and index publications, along with a new table that records all FERC company IDs and tracks how their reported name and other attributes change over time.
  • Format: The data will be distributed as a collection of Apache Parquet files, probably partitioned by quarter. All of the data from 2013Q3 to the present will use the same well-defined schema, informed by all of the historical EQR data dictionaries published by FERC.
  • Processing: Our first priority is making the bulk data freely accessible in a format that’s appropriate for analysis. We are not planning on doing any major data processing, beyond enforcing a uniform schema, converting the data to Parquet, and dealing with character encoding and CSV formatting issues that affect ~1% of all records.
  • Versioning: Due to the size of the data and the limited free cloud storage we have as part of the AWS Open Data Registry we won’t be able to provide multiple historical versions of the outputs like we do with other PUDL data. Whenever we update the outputs, the older version will be replaced.
  • Updates: We plan to update the outputs at least once a quarter, and will capture fresh snapshots of the raw EQR data using our automated data archiving scripts.
  • Access: We plan to publish the FERC EQR alongside the rest of our PUDL data in a freely accessible S3 bucket as part of the AWS Open Data Registry. We also intend to make the EQR outputs available for preview and querying through the PUDL Data Viewer, which will allow users to download smaller subsets of the data as CSVs for spreadsheet-based analysis if that’s what they need.

Get Involved!

  • If you’re already familiar with the FERC EQR and want to help test out the new data we’d love to hear from you: [email protected]
  • We’d like to better understand how this data is used in any context, whether it’s research, policy, journalistic, or commercial, so if you’ve got a use case in mind for historical electricity transaction data, let us know what it is. Maybe sign up for office hours so we can chat more, or help you get familiar with Parquet files.
  • What other data does the FERC EQR need be connected to? One of the things we’re hoping this analysis-ready, cloud-native version of EQR will enable is robust record linkage with other datasets. Can we identify the LMP nodes where power is being delivered? Can we identify electricity buyers in other FERC or EIA data, even though all we have is their name? What else should we be thinking about?
  • Stay up to date with the project: Subscribe to our ~monthly newsletter, keep track of the FERC EQR issues in the main PUDL repo, or follow the EQR scoping repo where we’ll be prototyping the new system.

After 5 years of ogling the EQR from afar, and projecting our (data) hopes and dreams into its depths, we’re excited to finally tackle it for real, and hope it’ll be a valuable resource for understanding US energy markets and accelerating the ever more economical decarbonization of the US electricity system.

Categories
updates

VCE RARE Renewable Generation Profiles in PUDL

Catalyst is helping GridLab, Pattern Energy Group, and Vibrant Clean Energy distribute a new open (CC-BY-4.0 licensed) dataset produced by Vibrant Clean Energy which provides hourly, county-level wind and solar generation profiles based on NOAA’s High Resolution Rapid Refresh (HRRR) weather model. The data release was announced at the ESIG’s Fall Technical Workshop on October 21st, 2024.

The new data is included in PUDL v2024.10.0 in Apache Parquet format. You can explore the dataset in this Jupyter notebook on Kaggle. The original 8760 hourly data as CSVs can be downloaded from Zenodo as well.

To learn more about why this kind of data is vital to the energy transition, check out ESIG’s report on Weather Data for Power System Planning.

VCE RARE Press Release

VCE RARE Power Dataset Press ReleaseDownload
Categories
updates

Announcing: Open Energy Data for All!

A free training series for energy graduate students

In the course of our work at Catalyst, we’ve been lucky enough to work with energy researchers at many institutions. So often, we see people run into the same set of problems – handling data that’s too big for Excel, reproducibly connecting datasets without shared IDs, and writing code that can be easily re-run and updated when a paper is reviewed or when a new year of data comes in.

Now, thanks to generous funding from the Alfred P. Sloan Foundation Energy & Environment Program, we’re pleased to announce “Open Energy Data for All,” an initiative to support energy graduate students. We’ll offer training in foundational data and software skills that enable faster, more open, and more reproducible energy data analysis.

Here’s what we have planned:

  • We’re hosting a monthly online seminar series addressing key challenges in energy data analysis
  • We’re working to develop a hands-on energy data curriculum with support from The Carpentries
  • We’ll give open energy data tutorials at several conferences around the country
  • Finally, we’ll host a two-day, in-person data lab, bringing together graduate students from across the U.S. to collectively tackle real-world energy data problems.
Kicking off our monthly seminar series

Whether wrestling with APIs or just trying to find the right dataset for your research, it can be hard to know where to start. That’s why, for our first monthly skill-building webinar, we’re starting with an introduction to the US open energy data landscape. Join us!

Intro to the US Open Energy Data Landscape
Oct. 30th, 4:30 – 5:30 PM Eastern (20:30 – 21:30 UTC)

Register for the Free Seminar

We’ll address questions like:

  • What free and public energy data exists for the U.S., and how can I access it?
  • How have people used that open data in the energy transition?
  • What common challenges are there with using this data?
  • How can I evaluate open datasets I find online?
  • How may data availability affect my research topics?

We’ll be running more webinars monthly – they’re designed to be drop-in, and will be recorded, so don’t worry too much about missing one or another.

Want to learn more?

If you’re interested in hearing more about these projects, sign up for our workshop newsletter and subscribe to our calendar of events! If you’re a faculty member interested in learning more, or in hosting us for a talk or workshop, we’d love to chat – drop us a line at [email protected].

Join the Workshop Email List
Workshop Events Calendar
Categories
updates

Workplace Democracy and Open Source

This year at csv,conf,v8 in Puebla, Mexico I gave a talk on our experience as a democratic worker cooperative creating digital public goods, and why we think co-ops are potentially a good fit for creating public-interest technology. You can watch the recorded talk on YouTube, or read on for a bloggified version of the talk below.

Continue reading “Workplace Democracy and Open Source”
Categories
updates

Integrating PUDL with PyPSA-USA

We recently found out that Kamran Tehranchi, one of two primary maintainers of the PyPSA-USA open source power system model, was working on adapting it to use open data that we publish through our Public Utility Data Liberation Project (PUDL), so we interviewed him over email to find out more about his experience making the switch.

Can you tell us a little bit about yourself? What problems are you working on? Where are you at?

Sure! I’m currently a PhD Student at Stanford University working in the Interdisciplinary Energy Systems (INES) Lab. By way of my research, I am also an energy system modeler and open-source software developer. My work focuses on electricity system planning, specifically on the impact of electricity transmission resolution within planning models. I primarily work with engineering-economic simulation and optimization models, mainly production cost simulations and capacity expansion models. I use these models to design and simulate future energy systems to understand the impacts of emerging technologies, policies, and climate-energy system interactions. One of the main projects I’ve been working on this past year is the PyPSA-USA planning model which in-part leverages PUDL to develop the electricity system data model.

Continue reading “Integrating PUDL with PyPSA-USA”
Categories
updates

Entity matching with Splink to connect FERC to EIA

Linking power plant financial data to energy system operational data with help from Climate Change AI

At the end of 2023, Catalyst wrapped up work funded by a Climate Change AI (CCAI) Innovation Grant using entity matching (record linkage) to connect the energy system financial data reported to the US Federal Energy Regulatory Commission (FERC) and physical energy system data reported to the US Energy Information Administration (EIA). While the data published in FERC Form 1 refers to the same utilities, power plants, and generators that are reported by EIA, these entities lack common IDs to link them. This connection between datasets is necessary to show that retiring certain fossil fuel power plants in favor of renewable energy sources is economically beneficial and technically feasible while still meeting the physical demands of today’s grid. Conducting entity matching to model this connection eliminates the extremely laborious process of sifting through these datasets and performing a manual connection. In collaboration with and support of RMI’s Utility Transition Hub, Catalyst created a small validation dataset of manually linked records, and thus know first hand the tedium of conducting this linkage manually.

Over the course of the grant period we developed the connection of FERC Form 1 plants to EIA data from a one-off module to an integrated analysis maintained and deployed with our nightly PUDL builds. Along the way, we updated our FERC-FERC plant connection (the plant_id_ferc1 column in out_ferc1__yearly_all_plants in the PUDL database), providing a unique plant ID to link FERC plants across all years of reporting. We believe our published output table of connections (out_pudl__yearly_assn_eia_ferc1_plant_parts in the PUDL database) is the only regularly updated, free and open-source connection between the FERC and EIA datasets. 

We hope the result enables advocates working to decarbonize our electricity system to more easily bring defensible and data-driven analyses to state-level legislative and regulatory processes. Additionally, we hope that the published matching framework can serve as an open-source example of record linkage for energy datasets and be a model for attempting similar connections with other energy datasets.

Inputs

The data published in FERC Form 1 is messy; reported records correspond to an assortment of generator aggregations (e.g. prime mover, primary fuel source, technology type, plants, or generator units). To create an EIA input that could match the diversity of records reported in FERC Form 1, we created the EIA “plant parts table”. This table contains aggregations of all EIA “plant parts” corresponding to the various granularities appearing in the FERC data.

FERC Input: out_ferc1__yearly_all_plants

EIA Input: out_eia__yearly_plant_parts

Model

After experimenting with several machine learning packages, we decided to use the open-source Python package Splink as it provided helpful transparency into the effects of changing model parameters and produced results better than our existing baseline. Splink is an entity matching and deduplication interface based on the Fellegi-Sunter algorithm for record linkage. Its main advantages are its speed working with data locally, its interface for users to define fuzzy matching logic between attributes in the input datasets, and its features for doing an unsupervised match (with no training data). Splink provides interactive charts of the model weights that make it easier for downstream users to provide feedback without advanced understanding of the underlying model mechanics.

Results

We used the manually matched dataset to evaluate the model results by a metric of precision and recall. Consider the set of FERC records in this manual validation dataset that the model predicted a matching EIA record for. Precision is the percentage of these matches that are correct. It represents the model’s accuracy when making a prediction. Now, consider all of the FERC records in the manual validation dataset. Recall is the percentage of these FERC records that the model predicted an EIA match for. It represents the model’s coverage of the FERC dataset. The table below displays the precision and recall of the Splink model alongside a baseline linear regression model that was previously integrated into PUDL. The “match probability threshold” is the threshold at which pairs with a lower probability of matching are labeled as a non-match. As the match threshold decreases, more record pairs are labeled as a match and the recall increases. However, precision decreases as the match threshold decreases because the match quality is lower and more FERC records are matched to an incorrect EIA record. Considering the needs of downstream users, we prioritized publishing match results with high precision and thus chose a match threshold of .9 for use in our deployed model.

Match Probability ThresholdPrecisionRecall
.95.944.833
.9.943.843
.75.940.862
.5.939.875
.25.938.887
baseline.90.73

Challenges and Limitations

One of the initial challenges we encountered during the project was the high percentage of null values in the input datasets. This significantly impacted the quality of our entity matching results. Additionally, our manually compiled training/validation dataset was relatively small and inherently introduced some unknown biases within the small sample size. Recognizing the dynamic nature of data over time and the potential shifts in representation as more data is published, we additionally experimented with the unsupervised training features for the Splink model. Results were similar to those of the supervised model, and we anticipate using the unsupervised model in the event that the existing training data becomes too outdated or fails to represent evolving patterns in the data. This forward-looking approach ensures adaptability to new data trends and optimizes for scenarios not adequately represented in the initial training dataset.

What’s Next?

With the development of this framework for entity matching, Catalyst is capable of greater flexibility and efficiency in data-driven model development. In 2024, we are building on this framework using funding from the Mozilla Foundation to link Security Exchange Commission utility ownership data to EIA utility operational data. We hope to leverage these models to address analogous issues in natural gas data in the future.

Catalyst is making exciting progress in providing open data to electricity resource planning models like the GridPath RA Toolkit with support from GridLab. Our initial work on these inputs has revealed that there is a need for entity matching in almost all of the datasets under consideration. For example, the Western Electricity Coordinating Council’s Reliability Modeling Anchor Data Set (WECC ADS) has transmission node IDs, generator IDs, and utility IDs that do not match other data sets referring to the same entities. We are excited to utilize the resource efficiency, usability, and transparency of Splink in building entity matching models for these datasets.

Please reach out to us with questions about the modeling process or resulting connection table, and let us know how you are utilizing the FERC to EIA connection!

Categories
updates

Beating the Utility Holding Company Shell Game

We’re excited to be part of the Mozilla Technology Fund’s 2024 cohort, which is focusing on open source AI for environmental justice!

We’re going to use Mozilla’s support to link US Securities and Exchange Commission data about utility ownership to financial and operational information in the EIA forms 860/861/923, and through our previous record linkage work involving the EIA data, to FERC Form 1 respondents and the EPA’s continuous emissions monitoring system data.

The SEC Form 10-K is published through EDGAR as structured XBRL data, but the Exhibit 21 attachment that describes which companies own and are owned by other companies is unfortunately just a PDF blob that gets stapled to the XBRL, and so ownership relationships end up being unstructured, or at best, semi-structured data.

We’re going to apply document modeling tools that we’ve developed in some of our client work (to extract structured data from PUC and other regulatory filings) to extract the ownership information from Exhibit 21. This will hopefully include the ownership percentages when they are reported.

Then we’re going to use the generalized entity matching / record linkage tooling that we developed under our previous Climate Change AI Innovation Grant to connect the parent / subsidiary companies named in the SEC data to the financial and operational data reported by the same utility companies in FERC Form 1, as well as EIA and EPA data.

The record linkage / entity matching system that we’ve ultimately settled on is based on the excellent (and publicly funded!) Splink library, which relies on DuckDB to enable local linkages on datasets of up to tens of millions of records. Robin Linacre (one of the Splink maintainers) has a tutorial explaining the probabilistic model of record linkage used by Splink, if you’re interested in the internals.

Why is this work important? Being able to make effective energy policy often requires an understanding of the political economy of utilities, and utilities are often composed of Russian doll-like nested holding companies. It can be hard to see where one utility ends and another begins. Understanding which entities share ownership and thus political and economic interests is key to being able to grapple with and influence them.

We’ll be learning from prior work on this problem done by the folks at CorpWatch, and we hope to make the outputs of our work easy to visualize and explore through the Oligrapher interface that LittleSis has developed.

If this work is interesting or useful to you, we’d love to hear more about your use case! You can track our work through this GitHub repository. Also, while we are explicitly focused on and familiar with utilities, the SEC’s Form 10-K covers all publicly traded companies, so we may be producing additional data outputs that aren’t useful to us but which could be useful to others. If that’s you, please let us know.

Categories
updates

You Don’t Have to Install PUDL Anymore

We’re excited to announce that you no longer have to install the PUDL Python library to access electric generation data linked across FERC and EIA such as capacity factor, heat rate, and fuel cost. These, and many others, are now available directly in the PUDL database, which you can download from Zenodo here. You can find more details on how to access the data here.

We were able to complete this large infrastructural overhaul with the help of generous funding from the Sloan foundation.

Now that you can use any tools you want to analyze the data, here are some ideas:

  • Use the same type of Python code you have been using, but freed from our tangled web of dependencies!
  • Use another language you like better: R, Rust, Ruby, or even other languages that don’t start with R (Julia?)
  • Use Kaggle to check out our data without installing any programming environments at all!
  • Hook up a BI tool to quickly generate low/no-code dashboards and visualizations!

Since we’re moving away from downstream use of the library, we are also deprecating the PudlTabl class. It will still work, for now, but it’s now just a shell around accessing the database tables and will be removed in a future release.

One further change we made during all of this was to rename a bunch of tables to make them a little easier to find and understand. Tables now have standardized prefixes, the nuances of which are explained in the docs. The short version is:

  • When in doubt, start with tables with the out_* prefix. These have been cleaned and connected into wide tables with lots of metadata and are designed to be easy to use for downstream analysis.
  • When you need to dig deeper, look at the core_* tables. These are the cleaned up building blocks of the out_* tables. You may need to join several core_* tables to get the metadata you want.
  • The tables starting with an underscore are intermediate assets. They’re not stable, so please don’t rely on the data in them.

We hope these changes make it easier for a wider variety of users to use our data! Now that we’ve wrapped up this infrastructural work, we’ll shift our focus back to integrating new datasets like PHMSA and EIA 176.

If you want help getting started with our data, or have any datasets you’d like us to integrate, we’d love to talk: drop by our office hours and we’ll walk you through any questions you might have.