This spreadsheet contains information reported by over 200 investor-owned utilities to the Federal Energy Regulatory Commission in the annual filing FERC Form 1 for the years 1994-2019. It contains 1) annual capital costs for new transmission, distribution, and administrative infrastructure; 2) annual operation and maintenance costs for transmission, distribution, and utility business administration; 3) total annual MWh sales and sales by customer class; 4) annual peak demand in MW; and 5) total customer count and the number of customers by class. Annual spending on new capital infrastructure is read from pages 204 to 207 of FERC Form 1, titled Electric Plant in Service. Annual transmission capital additions are recorded from Line 58, Column C - Total Transmission Plant Additions. Likewise, annual distribution capital additions are recorded from Line 75, Column C - Total Distribution Plant Additions. Administrative capital additions are recorded from Line 5, Column C - Total Intangible Plant Additions, and Line 99, Column C - Total General Plant Additions. Operation and maintenance costs associated with transmission, distribution, and utility administration are read from pages 320 to 323 of FERC Form 1, titled Electric Operation and Maintenance Expenses. Annual transmission operation and maintenance are recorded from Line 99, Column B - Total Transmission Operation Expenses for Current Year, and Line 111, Column B - Total Transmission Maintenance Expenses for Current Year. Likewise, annual distribution operation and maintenance costs are recorded from Line 144, Column B - Total Distribution Operation Expenses, and Line 155, Column B - Total Distribution Maintenance Expenses. Administrative operation and maintenance costs are recorded from: Line 164, Column B - Total Customers Accounts Expenses; Line 171, Column B - Total Customer Service and Information Expenses; Line 178, Column B - Total Sales Expenses; and Line 197, Column B - Total Administrative and General Expenses. The annual peak demand in MW over the year is read from page 401, titled Monthly Peaks and Output. The monthly peak demand is listed in Lines 29 to 40, Column D. The maximum of these monthly reports during each year is taken as the annual peak demand in MW. The annual energy sales and customer count data come from page 300, Electric Operating Revenues. The values are provided in Line 2 - Residential Sales, Line 4 - Commercial Sales, Line 5 - Industrial Sales, and Line 10 - Total Sales to Ultimate Consumers. More information about the database is available in an associated report published by the University of Texas at Austin Energy Institute: https://live-energy-institute.pantheonsite.io/sites/default/files/UTAustin_FCe_TDA_2016.pdf Also see an associated paper published in the journal Energy Policy: Fares, Robert L., and Carey W. King. "Trends in transmission, distribution, and administration costs for US investor-owned electric utilities." Energy Policy 105 (2017): 354-362. https://doi.org/10.1016/j.enpol.2017.02.036 All data come from the Federal Energy Regulatory Commission FERC Form 1 Database available in Microsoft Visual FoxPro Format: https://www.ferc.gov/docs-filing/forms/form-1/data.asp

DOLFORMS Dataset contains the most frequently requested Department of Labor forms. More information and details about the data provided can be found at http://webapps.dol.gov/libraryforms/FormsByAgency.asp

This is the first data release from the Public Utility Data Liberation (PUDL) project. It can be referenced & cited using https://doi.org/10.5281/zenodo.3653159

For more information about the free and open source software used to generate this data release, see Catalyst Cooperative's PUDL repository on Github, and the associated documentation on Read The Docs. This data release was generated using v0.3.1 of the catalystcoop.pudl python package.

Included Data Packages

This release consists of three tabular data packages, conforming to the standards published by Frictionless Data and the Open Knowledge Foundation. The data are stored in CSV files (some of which are compressed using gzip), and the associated metadata is stored as JSON. These tabular data can be used to populate a relational database.

• pudl-eia860-eia923:
  Data originally collected and published by the US Energy Information Administration (US EIA). The data from EIA Form 860 covers the years 2011-2018. The Form 923 data covers 2009-2018. A large majority of the data published in the original data sources has been included, but some parts, like fuel stocks on hand, and EIA 923 schedules 6, 7, & 8 have not yet been integrated.
• pudl-eia860-eia923-epacems:
  This data package contains all of the same data as the pudl-eia860-eia923 package above, as well as the Hourly Emissions data from the US Environmental Protection Agency's (EPA's) Continuous Emissions Monitoring System (CEMS) from 1995-2018. The EPA CEMS data covers thousands of power plants at hourly resolution for decades, and contains close to a billion records.
• pudl-ferc1:
  Seven data tables from FERC Form 1 are included, primarily relating to individual power plants, and covering the years 1994-2018 (the entire span of time for which FERC provides this data). These tables are the only ones which have been subjected to any cleaning or organization for programmatic use within PUDL. The complete, raw FERC Form 1 database contains 116 different tables with many thousands of columns of mostly financial data. We will archive a complete copy of the multi-year FERC Form 1 Database as a file-based SQLite database at Zenodo, independent of this data release. It can also be re-generated using the catalystcoop.pudl Python package and the original source data files archived as part of this data release.

Contact Us

If you're using PUDL, we would love to hear from you! Even if it's just a note to let us know that you exist, and how you're using the software or data. You can also:

• Subscribe to our announcements list for email updates.
• Use the Github issue tracker to file bugs, suggest improvements, or ask for help.
• Email the project team at pudl@catalyst.coop for private communications.
• Follow @CatalystCoop on Twitter.

Using the Data

The data packages are just CSVs (data) and JSON (metadata) files. They can be used with a variety of tools on many platforms. However, the data is organized primarily with the idea that it will be loaded into a relational database, and the PUDL Python package that was used to generate this data release can facilitate that process. Once the data is loaded into a database, you can access that DB however you like.

Make sure conda is installed

None of these commands will work without the conda Python package manager installed, either via Anaconda or miniconda:

• Install Anaconda
• Install miniconda

Download the data

First download the files from the Zenodo archive into a new empty directory. A couple of them are very large (5-10 GB), and depending on what you're trying to do you may not need them.

• If you don't want to recreate the data release from scratch by re-running the entire ETL process yourself, and you don't want to create a full clone of the original FERC Form 1 database, including all of the data that has not yet been integrated into PUDL, then you don't need to download pudl-input-data.tgz.
• If you don't need the EPA CEMS Hourly Emissions data, you do not need to download pudl-eia860-eia923-epacems.tgz.

Load All of PUDL in a Single Line

Use cd to get into your new directory at the terminal (in Linux or Mac OS), or open up an Anaconda terminal in that directory if you're on Windows.

If you have downloaded all of the files from the archive, and you want it all to be accessible locally, you can run a single shell script, called load-pudl.sh:

bash pudl-load.sh

This will do the following:

• Load the FERC Form 1, EIA Form 860, and EIA Form 923 data packages into an SQLite database which can be found at sqlite/pudl.sqlite.
• Convert the EPA CEMS data package into an Apache Parquet dataset which can be found at parquet/epacems.
• Clone all of the FERC Form 1 annual databases into a single SQLite database which can be found at sqlite/ferc1.sqlite.

Selectively Load PUDL Data

If you don't want to download and load all of the PUDL data, you can load each of the above datasets separately.

Create the PUDL conda Environment

This installs the PUDL software locally, and a couple of other useful packages:

conda create --yes --name pudl --channel conda-forge \
  --strict-channel-priority \
  python=3.7 catalystcoop.pudl=0.3.1 dask jupyter jupyterlab seaborn pip
conda activate pudl

Create a PUDL data management workspace

Use the PUDL setup script to create a new data management environment inside this directory. After you run this command you'll see some other directories show up, like parquet, sqlite, data etc.

pudl_setup ./

Extract and load the FERC Form 1 and EIA 860/923 data

If you just want the FERC Form 1 and EIA 860/923 data that has been integrated into PUDL, you only need to download pudl-ferc1.tgz and pudl-eia860-eia923.tgz. Then extract them in the same directory where you ran pudl_setup:

tar -xzf pudl-ferc1.tgz
tar -xzf pudl-eia860-eia923.tgz

To make use of the FERC Form 1 and EIA 860/923 data, you'll probably want to load them into a local database. The datapkg_to_sqlite script that comes with PUDL will do that for you:

datapkg_to_sqlite \
  datapkg/pudl-data-release/pudl-ferc1/datapackage.json \
  datapkg/pudl-data-release/pudl-eia860-eia923/datapackage.json \
  -o datapkg/pudl-data-release/pudl-merged/

Now you should be able to connect to the database (~300 MB) which is stored in sqlite/pudl.sqlite.

Extract EPA CEMS and convert to Apache Parquet

If you want to work with the EPA CEMS data, which is much larger, we recommend converting it to an Apache Parquet dataset with the included epacems_to_parquet script. Then you can read those files into dataframes directly. In Python you can use the pandas.DataFrame.read_parquet() method. If you need to work with more data than can fit in memory at one time, we recommend using Dask dataframes. Converting the entire dataset from datapackages into Apache Parquet may take an hour or more:

tar -xzf pudl-eia860-eia923-epacems.tgz
epacems_to_parquet datapkg/pudl-data-release/pudl-eia860-eia923-epacems/datapackage.json

You should find the Parquet dataset (~5 GB) under parquet/epacems, partitioned by year and state for easier querying.

Clone the raw FERC Form 1 Databases

If you want to access the entire set of original, raw FERC Form 1 data (of which only a small subset has been cleaned and integrated into PUDL) you can extract the original input data that's part of the Zenodo archive and run the ferc1_to_sqlite script using the same settings file that was used to generate the data release:

tar -xzf pudl-input-data.tgz
ferc1_to_sqlite data-release-settings.yml

You'll find the FERC Form 1 database (~820 MB) in sqlite/ferc1.sqlite.

Data Quality Control

We have performed basic sanity checks on much but not all of the data compiled in PUDL to ensure that we identify any major issues we might have introduced through our processing

Protein-Protein, Genetic, and Chemical Interactions for Pineda AO (2004):The anticoagulant thrombin mutant W215A/E217A has a collapsed primary specificity pocket. curated by BioGRID (https://thebiogrid.org); ABSTRACT: The thrombin mutant W215A/E217A features a drastically impaired catalytic activity toward chromogenic and natural substrates but efficiently activates the anticoagulant protein C in the presence of thrombomodulin. As the remarkable anticoagulant properties of this mutant continue to be unraveled in preclinical studies, we solved the x-ray crystal structures of its free form and its complex with the active site inhibitor H-d-Phe-Pro-Arg-CH(2)Cl (PPACK). The PPACK-bound structure of W215A/E217A is identical to the structure of the PPACK-bound slow form of thrombin. On the other hand, the structure of the free form reveals a collapse of the 215-217 strand that crushes the primary specificity pocket. The collapse results from abrogation of the stacking interaction between Phe-227 and Trp-215 and the polar interactions of Glu-217 with Thr-172 and Lys-224. Other notable changes are a rotation of the carboxylate group of Asp-189, breakage of the H-bond between the catalytic residues Ser-195 and His-57, breakage of the ion pair between Asp-222 and Arg-187, and significant disorder in the 186- and 220-loops that define the Na(+) site. These findings explain the impaired catalytic activity of W215A/E217A and demonstrate that the analysis of the molecular basis of substrate recognition by thrombin and other proteases requires crystallization of both the free and bound forms of the enzyme.

Regulation (EC) No 259/2008 required Member States to publish the amounts of aid received by each beneficiary under the CAP. Following the judgment of the Court of Justice of the European Union delivered on 9 November 2010, the legal basis for the publication of beneficiaries of CAP payments was partially invalidated: it now concerns only legal persons.The data available through Etalab are therefore for each corporate form concerned, the amounts collected under the EAFRD on the one hand (excluding and included subsidised loans) and the EAGGF (direct and other payments) on the other hand. Source:Service and Payment Agency (ASP)The data available through Etalab are therefore for each corporate form concerned, the amounts collected under the EAFRD on the one hand (excluding and included subsidised loans) and the EAGGF (direct and other payments) on the other hand. Source: Service and Payment Agency (ASP)