Raw US Energy Information Administration (EIA) Form 861 Data, archived from https://www.eia.gov/electricity/data/eia861/

This archive contains raw input data for the Public Utility Data Liberation (PUDL) software developed by Catalyst Cooperative. It is organized into Frictionless Data Packages. For additional information about this data and PUDL, see the following resources:

The PUDL Repository on GitHub

PUDL Documentation

Other Catalyst Cooperative data archives

Annual data on electricity generating capacity, electricity generation and useful thermal output, fuel receipts, fuel stocks, sales, consumption, and emissions in the United States. Based on Form EIA-861 and Form EIA-860 data. Annual time series extend back to 1994.

Hourly Electricity Demand by State

This archive contains the output of the Public Utility Data Liberation (PUDL) Project state electricity demand allocation analysis, as of the v0.4.0 release of the PUDL Python package. Here is the script that produced this output. It was run using the Docker container and processed data that are included in PUDL Data Release v2.0.0.

The analysis uses hourly electricity demand reported at the balancing authority and utility level in the FERC 714 (data archive), and service territories for utilities and balancing authorities inferred from the counties served by each utility, and the utilities that make up each balancing authority in the EIA 861 (data archive), to estimate the total hourly electricity demand for each US state.

We used the total electricity sales by state reported in the EIA 861 as a scaling factor to ensure that the magnitude of electricity sales is roughly correct, and obtains the shape of the demand curve from the hourly planning area demand reported in the FERC 714. The scaling is necessary partly due to imperfections in the historical utility and balancing authority service territory maps which we have been able to reconstruct from the data reported in the EIA 861 Service Territories and Balancing Authority tables.

The compilation of historical service territories based on the EIA 861 data is somewhat manual and could be improved, but overall the results seem reasonable. Additional predictive spatial variables will be required to obtain more granular electricity demand estimates (e.g. at the county level).

FERC 714 Respondents

The file ferc714_respondents.csv links FERC Form 714 respondents to what we believe to be their corresponding EIA utilities or balancing authorities.

• eia_code: An integer ID reported in the FERC Form 714 corresponding to the respondent's EIA ID. In some cases this is a Utility ID, and in others it is a Balancing Authority ID, but which is not specified and so we have had to infer the type of entity which is responding. Note that in many cases the same company acts as both a utility and a balancing authority, and the integer ID associated with the company is often the same in both roles, but it does not need to be.
• respondent_type: Either balancing_authority or utility depending on which type of entity we believe was responding to the FERC 714.
• respondent_id_ferc714: The integer ID of the responding entity within the FERC 714.
• respondent_name_ferc714: The name provided by the respondent in the FERC 714.
• balancing_authority_id_eia: If the respondent was identified as a balancing authority, the EIA ID for that balancing authority, taken from the EIA Form 861.
• balancing_authority_code_eia: If the respondent was identified as a balancing authority, the EIA short code used to identify the balancing authority, taken from the EIA Form 861.
• balancing_authority_name_eia: If the respondent was identified as a balancing authority, the name of the balancing authority, taken from the EIA Form 861.
• utility_id_eia: If the respondent was identified as a utility, the EIA utility ID, taken from the EIA Form 861.
• utility_name_eia: If the respondent was identified as a utility, the name of the utility, taken from the EIA 861.

FERC 714 Respondent Service Territories

The file ferc714_service_territories.csv describes the historical service territories for FERC 714 respondents for the years 2006-2019. For each respondent and year, their service territory is composed of a collection of counties, identified by their 5-digit FIPS codes. The file contains the following columns, with each row associating a single county with a FERC 714 respondent in a particular year:

• respondent_id_ferc714: The FERC Form 714 respondent ID, which is also found in ferc714_respondents.csv
• report_date: The first day of the year for which the service territory is being described.
• state: Two letter abbreviation for the state containing the county, for human readability.
• county: The name of the county, for human readability.
• state_id_fips: The 2-digit FIPS state code.
• county_id_fips: The 5-digit FIPS county code for use with other geospatial data resources, like the US Census DP1 geodatabase.

State Hourly Electricity Demand Estimates

The file demand.csv contains hourly electricity demand estimates for each US state from 2006-2019. It contains the following columns:

• state_id_fips: The 2-digit FIPS state code.
• utc_datetime: UTC time at hourly resolution.
• demand_mwh: Electricity demand for that state and hour in MWh. This is an allocation of the electricity demand reported directly in the FERC Form 714.
• scaled_demand_mwh: Estimated total electricity demand for that state and hour, in MWh. This is the reported FERC Form 714 hourly demand scaled up or down linearly such that the total annual electricity demand matches the total annual electricity sales reported at the state level in the EIA Form 861.

A collection of plots are also included, comparing the original and scaled demand time series for each state.

Acknowledgements

This analysis was funded largely by GridLab, and done in collaboration with researchers at the Lawrence Berkeley National Laboratory, including Umed Paliwal and Nikit Abhyankar.

• Ethan Welty wrote the final code and most of the algorithms.
• Yash Kumar did initial data explorations and geospatial analyses.

The data screening methods were originally designed to identify unrealistic data in the electricity demand timeseries reported to EIA on Form 930, and have been applied here to data form the FERC Form 714.

They are adapted from code published and modified by:

• Tyler Ruggles
• Greg Schivley

And described at:

• Developing reliable hourly electricity demand data through screening and imputation
• EIA Cleaned Hourly Electricity Demand Code (Zenodo)
• EIA Cleaned Hourly Electricity Demand Code (GitHub)

The imputation methods were designed for multivariate time series forecasting.

They are adapted from code published by:

• Xinyu Chen chenxy346@gmail.com

And described at:

• Low-Rank Autoregressive Tensor Completion for Multivariate Time Series Forecasting
• Scalable Low-Rank Tensor Learning for Spatiotemporal Traffic Data Imputation
• Tensor Learning (张量学习)

About PUDL & Catalyst Cooperative

For additional information about this data and PUDL, see the following resources:

• The PUDL Repository on GitHub
• The PUDL Documentation
• Other Catalyst Cooperative data archives on Zenodo

These files contain data that are submitted to the Energy Information Administration (EIA) by selected electric utilities on Form-826, "Monthly Electric Utility Sales and Revenue Report with State distributions." Form EIA-826 is used to collect retail sales of electricity (in megawatthours) and associated revenue (in thousands of dollars). Respondents are chosen from a sample of electric utilities in the United States that complete Form EIA-861, "Annual Electric Utility Report." (See CISER codebook number ECON-070).

This API provides data on retail sales of electricity by major end-use sectors, i.e., residential, commercial, industrial, and transportation. Based on Form EIA-826 and Form EIA-861 data. Annual, quarterly, and monthly data available.

Data on annual emissions of Carbon Dioxide (CO2), Sulfur Dioxide (SO2), and Nitrogen Oxides (NOx). Data organized by type of electric power producer, by energy source, and by U.S. state. Annual time series extend back to 1990. Based on Form EIA-861 data. Electric Power Producer: Commercial Cogen, Commercial Non-Cogen, Electric Utility, Industrial Cogen, Industrial Non-Cogen, IPP NAICS-22 Cogen, IPP NAICS-22 Non-Cogen, and Total Electric Power Industry Energy Source: Coal, Geothermal, Natural Gas, Other, Other Biomass, Other Gases, Wood and Wood Derived Fuels, Petroleum, and All Energy Sources

This data deposit constructs data on monthly generation costs and capacities in the United States from 1999-2012 in preparation for their use in "Imperfect Markets versus Imperfect Regulation in U.S. Electricity Generation" (openicpsr-115467).It builds panel data files from the following EIA forms:"Form EIA-860: Annual Electric Generator Report""Form EIA-861: Annual Electric Power Industry Report""Form EIA-767: Annual Steam-Electric Plant Operation and Design Report""Form EIA-923: Power Plant Operations Report""Form EIA-759/906/920/923: Power Plant Report""Form EIA-423: Monthly Cost and Quality of Fuels for Electric Plants Report"and the EPA's Continuous Emissions Monitor System.It constructs a crosswalk that connects the EPA's boilers to the EIA's generator identifiers.

The Annual Electric Utility Report, taken from Form EIA-861, collects information on the status of electric utilities and their generation, transmission, and distribution of electric energy in the United States, its territories, and Puerto Rico. The data from this form are used to accurately maintain the EIA electric utility frame, to draw samples for other electric power surveys, and to provide input for other EIA publications. The data include average revenue per kilowatthour, average monthly consumption, and operating revenue from sales.

Annual data back to 2003 at the national level for electricity generation; capacity; consumption and cost of fossil fuels; sales, price and revenue; emissions; demand-side management; and operating revenues, expenses, and income. Based on Form EIA-860 and Form EIA-861 data.

Annual data on the average price of retail electricity to consumers. Data organized by U.S. state and by provider, i.e., total electric industry, full-service providers, restructured retail service providers, energy-only providers, and delivery-only service. Annual time series extend back to 1990. Based on Form EIA-861 data.

These files contain data that are submitted to the Energy Information Administration (EIA) by selected electric utilities on Form-826, "Monthly Electric Utility Sales and Revenue Report with State distributions." Form EIA-826 is used to collect retail sales of electricity (in megawatthours) and associated revenue (in thousands of dollars). Respondents are chosen from a sample of electric utilities in the United States that complete Form EIA-861, "Annual Electric Utility Report." (See CISER codebook number ECON-070).

Data on sales, costs, rates, area, rates of return and equity, and illustrative bills for power providers in California. Source data documents and cleaned tables are included.

Datasets included: 1. CCA JRC PDFs: Joint Rate Comparisons for Community Choice Aggregators, scraped from IOU website via the Wayback Machine 2. EIA 861 Tables: collected EIA Form 861 datasets and associated datacleaning code 3. POU Financial Statements: annual audited financials of LADWP and SMUD 4. Residential and Commercial Rate Source PDFs: for IOUs and POUs, source files for rate time series 5. Appendix Data Part 1 - Rates, Usage, Area, Costs: 5a: Core CPI rates 5b: Residential and Commercial Rate time series 5c: IOU rate base 5d: IOU O&M 5e: POU utility plant 5f: POU O&M 5g: CCA rate comparisons 5h: detailed IOU O&M data 5i: consolidated EIA sales data (all CA LSEs) 5j: consolidated EIA sales data with average prices (selected CA and non-CA LSEs) 5k: CCA service area calculation 5l: interstate baseline residential rate comparison 5m: extended sample of CA POUs O&M and utility plant 6. Appendix Data Part 2 - Illustrative Monthly Bills: comparisons between CCA and IOUs, with customer load shape and intermediate calculations 7. Appendix Data Part 3 - ROE and ROR: time series of rates of return and return on equity for selected IOUs 8. PUDL FERC1 Processing Code: Python code to pull latest stable PUDL dataset and extract IOU O&M data as required

The U.S. Energy Information Administration complies this calendar-year data to show a comparison for high voltage service. Source: Form EIA-861 2016 Data.

PUDL v2024.2.6 Data Release

The main impetus behind this release is the quarterly update of some of ourcore datasets with preliminary data for 2023Q4. The EIA Form 860 – Annual Electric Generator Report, EPA Hourly Continuous Emission Monitoring System (CEMS), and bulk EIA API data are all up to date through the end of 2023, while the EIA Form 923 – Power Plant Operations Report lags a month behind and is currently only available through November, 2023. We also addressed several issues we found in our initial release automation process that will make it easier for us to do more frequent releases, like this one!

We’re also for the first time publishing the full historical time series of of generator data available in the EIA860M, rather than just using the most recent release to update the EIA860 outputs. This enables tracking of how planned fossil plant retirement dates have evolved over time.

There are also updates to our data validation system, a new version of Pandas, and experimental Parquet outputs. See below for the details.

New Data Coverage

• Add EIA860M data through December 2023 #3313, #3367.
• Add 2023 Q4 of CEMS data. See #3315, #3379.
• Add EIA923 monthly data through November 2023 #3314, #3398, #3422.
• Create a new table core_eia860m_changelog_generators which tracks the evolution of all generator data reported in the EIA860M, in particular the stated retirement dates. see issue #3330 and PR #3331. Previously only the most recent month of reported EIA860M data was available within the PUDL DB.

Release Infrastructure

• Use the same logic to merge version tags into the stable branch as we are using to merge the nightly build tags into the nightly branch. See PR #3347
• Automatically place a temporary object hold on all versioned data releases that we publish to GCS, to ensure that they can’t be accidentally deleted. See issue #3400 and PR #3421.

Schema Changes

Restored the individual FERC Form 1 plant output tables, providing direct access to denormalized versions of the specific plant types via:

• out_ferc1_yearly_steam_plants_sched402
• out_ferc1_yearly_small_plants_sched410
• out_ferc1_yearly_hydroelectric_plants_sched406
• out_ferc1_yearly_pumped_storage_plants_sched408

See issue #3416 & PR #3417

Data Validation with Pandera

We’ve started integrating pandera dataframe schemas and checks with dagster asset checks to validate data while our ETL pipeline is running instead of only after all the data has been produced. Initially we are using the various database schema checks that are generated by our metadata, but the goal is to migrate all of our data validation tests into this framework over time, and to start using it to encode any new data validations immediately. See issues #941, #1572, #3318, #3412 and PR #3282.

Pandas 2.2

We’ve updated to Pandas 2.2, which has a number of changes and deprecations. See PRs #3272, #3410.

• Changes in how merge results are sorted impacted the assignment of unit_id_pudl values, so any hard-coded values that dependent on the previous assignments will likely be incorrect now. We had to update a number of tests and FERC1-EIA record linkage training data to account for this change.
• Pandas is also deprecating the use of the AS frequency alias, in favor of YS, so many references to the old alias have been updated.
• We’ve switched to using the calamine engine for reading Excel files, which is much faster than the old openpyxl library.

Parquet Outputs

The ETL now outputs PyArrow Parquet files for all tables that are written to the PUDL DB. The Parquet outputs are used as the interim storage for the ETL, rather than reading all tables out of the SQLite DB. We aren’t publicly distributing the Parquet outputs yet, but are giving them a test run with some existing users. See #3102 #3296, #3399.

Other PUDL v2024.2.6 Resources

• PUDL v2024.2.6 Data Dictionary
• PUDL v2024.2.6 Documentation
• PUDL in the AWS Open Data Registry
• PUDL v2024.2.6 in a free, public AWS S3 bucket: s3://pudl.catalyst.coop/v2024.2.6/
• PUDL v2024.2.6 in a requester-pays GCS bucket: gs://pudl.catalyst.coop/v2024.2.6/
• Zenodo archive of the PUDL GitHub repo for this release
• PUDL v2024.2.6 release on GitHub
• PUDL v2024.2.6 package in the Python Package Index (PyPI)

Contact Us

If you're using PUDL, we would love to hear from you! Even if it's just a note to let us know that you exist, and how you're using the software or data. Here's a bunch of different ways to get in touch:

• Follow us on GitHub
• Use the PUDL Github issue tracker to let us know about any bugs or data issues you encounter
• GitHub Discussions is where we provide user support.
• Watch our GitHub Project to see what we're working on.
• Email us at hello@catalyst.coop for private communications.
• On Mastodon: @CatalystCoop@mastodon.energy
• On BlueSky: @catalyst.coop
• On Twitter: @CatalystCoop
• Play with our data and notebooks on Kaggle
• Combine our data with ML models on HuggingFace
• Learn more about us on our website: https://catalyst.coop
• Subscribe to our announcements list for email updates.

The U.S. Energy Information Administration complies this calendar-year data to show a comparison for commercial rates. Source: Form EIA-861 2016 Data.

Historical U.S. electric utility data. Data on generation, electric purchases, peak load, sales, revenues, customer counts, demand-side management programs, green pricing, net metering programs, and distributed generation capacity. Based on EIA Form-861 data. Data contained in a zip file.

The U.S. Energy Information Administration complies this calendar-year data to show a comparison for residential rates. Source: Form EIA-861 2016 Data.

Historical U.S. electric utility data. Data on generation, electric purchases, peak load, sales, revenues, customer counts, demand-side management programs, green pricing, net metering programs, and distributed generation capacity. Based on EIA Form-861 data. Data contained in a zip file.