This is the first data release from the Public Utility Data Liberation (PUDL) project. It can be referenced & cited using https://doi.org/10.5281/zenodo.3653159 For more information about the free and open source software used to generate this data release, see Catalyst Cooperative's PUDL repository on Github, and the associated documentation on Read The Docs. This data release was generated using v0.3.1 of the catalystcoop.pudl python package. Included Data Packages This release consists of three tabular data packages, conforming to the standards published by Frictionless Data and the Open Knowledge Foundation. The data are stored in CSV files (some of which are compressed using gzip), and the associated metadata is stored as JSON. These tabular data can be used to populate a relational database. pudl-eia860-eia923: Data originally collected and published by the US Energy Information Administration (US EIA). The data from EIA Form 860 covers the years 2011-2018. The Form 923 data covers 2009-2018. A large majority of the data published in the original data sources has been included, but some parts, like fuel stocks on hand, and EIA 923 schedules 6, 7, & 8 have not yet been integrated. pudl-eia860-eia923-epacems: This data package contains all of the same data as the pudl-eia860-eia923 package above, as well as the Hourly Emissions data from the US Environmental Protection Agency's (EPA's) Continuous Emissions Monitoring System (CEMS) from 1995-2018. The EPA CEMS data covers thousands of power plants at hourly resolution for decades, and contains close to a billion records. pudl-ferc1: Seven data tables from FERC Form 1 are included, primarily relating to individual power plants, and covering the years 1994-2018 (the entire span of time for which FERC provides this data). These tables are the only ones which have been subjected to any cleaning or organization for programmatic use within PUDL. The complete, raw FERC Form 1 database contains 116 different tables with many thousands of columns of mostly financial data. We will archive a complete copy of the multi-year FERC Form 1 Database as a file-based SQLite database at Zenodo, independent of this data release. It can also be re-generated using the catalystcoop.pudl Python package and the original source data files archived as part of this data release. Contact Us If you're using PUDL, we would love to hear from you! Even if it's just a note to let us know that you exist, and how you're using the software or data. You can also: Subscribe to our announcements list for email updates. Use the Github issue tracker to file bugs, suggest improvements, or ask for help. Email the project team at pudl@catalyst.coop for private communications. Follow @CatalystCoop on Twitter. Using the Data The data packages are just CSVs (data) and JSON (metadata) files. They can be used with a variety of tools on many platforms. However, the data is organized primarily with the idea that it will be loaded into a relational database, and the PUDL Python package that was used to generate this data release can facilitate that process. Once the data is loaded into a database, you can access that DB however you like. Make sure conda is installed None of these commands will work without the conda Python package manager installed, either via Anaconda or miniconda: Install Anaconda Install miniconda Download the data First download the files from the Zenodo archive into a new empty directory. A couple of them are very large (5-10 GB), and depending on what you're trying to do you may not need them. If you don't want to recreate the data release from scratch by re-running the entire ETL process yourself, and you don't want to create a full clone of the original FERC Form 1 database, including all of the data that has not yet been integrated into PUDL, then you don't need to download pudl-input-data.tgz. If you don't need the EPA CEMS Hourly Emissions data, you do not need to download pudl-eia860-eia923-epacems.tgz. Load All of PUDL in a Single Line Use cd to get into your new directory at the terminal (in Linux or Mac OS), or open up an Anaconda terminal in that directory if you're on Windows. If you have downloaded all of the files from the archive, and you want it all to be accessible locally, you can run a single shell script, called load-pudl.sh: bash pudl-load.sh This will do the following: Load the FERC Form 1, EIA Form 860, and EIA Form 923 data packages into an SQLite database which can be found at sqlite/pudl.sqlite. Convert the EPA CEMS data package into an Apache Parquet dataset which can be found at parquet/epacems. Clone all of the FERC Form 1 annual databases into a single SQLite database which can be found at sqlite/ferc1.sqlite. Selectively Load PUDL Data If you don't want to download and load all of the PUDL data, you can load each of the above datasets separately. Create the PUDL conda Environment This installs the PUDL software locally, and a couple of other u...

The open data portal catalogue is a downloadable dataset containing some key metadata for the general datasets available on the Government of Canada's Open Data portal. Resource 1 is generated using the ckanapi tool (external link) Resources 2 - 8 are generated using the Flatterer (external link) utility. ###Description of resources: 1. Dataset is a JSON Lines (external link) file where the metadata of each Dataset/Open Information Record is one line of JSON. The file is compressed with GZip. The file is heavily nested and recommended for users familiar with working with nested JSON. 2. Catalogue is a XLSX workbook where the nested metadata of each Dataset/Open Information Record is flattened into worksheets for each type of metadata. 3. datasets metadata contains metadata at the dataset level. This is also referred to as the package in some CKAN documentation. This is the main table/worksheet in the SQLite database and XLSX output. 4. Resources Metadata contains the metadata for the resources contained within each dataset. 5. resource views metadata contains the metadata for the views applied to each resource, if a resource has a view configured. 6. datastore fields metadata contains the DataStore information for CSV datasets that have been loaded into the DataStore. This information is displayed in the Data Dictionary for DataStore enabled CSVs. 7. Data Package Fields contains a description of the fields available in each of the tables within the Catalogue, as well as the count of the number of records each table contains. 8. data package entity relation diagram Displays the title and format for column, in each table in the Data Package in the form of a ERD Diagram. The Data Package resource offers a text based version. 9. SQLite Database is a .db database, similar in structure to Catalogue. This can be queried with database or analytical software tools for doing analysis.

SECTIC-24K is a digital file of the Public Land Survey (PLS) section corners of Minnesota as recorded on the U.S. Geological Survey's 1:24,000 7.5-minute quadrangle maps (map dates ranging from the late 1940s - 1970s). The database attempts to best fit the section corner locations shown on the published 1:24,000 maps, even though better real-world data for the location of the section corner might be available elsewhere. The SECTIC-24K data set also includes a program which has the following utilities:

Utility A: Section corner extraction from the SECTIC-24K database by county, 1:24,000-scale quad, or township.
Utility B: Conversion among PLS, UTM, or LAT/LONG coordinates, either interactively or by file conversion. It also allows NAD27 - NAD83 conversions.
Utility C: Creation of a dBASE output file from SECTIC-24K.

This program does not run on Windows 7 - 64 bit computers (it does run on Windows - 32 bit). There is also a web service that generates much the same info as the SECTIC program. The main differences are it may not do NAD27/NAD83 shifts and it doesn't have a batch mode. A batch mode could be created using the web service and the scripting code of your choice. Find the web service at: https://gisdata.mn.gov/dataset/loc-pls-api-service

ARGN17.WDM contains nine data series: air temperature in degrees Fahrenheit (dsn 400), dewpoint temperature in degrees Fahrenheit (dsn 500), wind speed in miles per hour (dsn 300), solar radiation in Langleys (dsn 600), computed potential evapotranspiration in thousandths of an inch (dsn 200), and four data-source flag series for air temperature (dsn 410), dewpoint temperature (dsn 510), wind speed (dsn 310) and solar radiation (dsn 610) respectively from January 1,1948, to September 30, 2017. The primary source of the data is Argonne National Laboratory (Argonne National Laboratory, 2017) and is processed following the guidelines documented in Over and others (2010). Daily potential evapotranspiration (PET) in thousandths of an inch is computed from average daily air temperature in degrees Fahrenheit (°F), average daily dewpoint temperature in degrees Fahrenheit (°F), daily total wind movement in miles (mi), and daily total solar radiation in Langleys per day (Lg/d) and disaggregated to hourly PET in thousandths of an inch using the Fortran program LXPET (Murphy, 2005). Missing and apparently erroneous data values were replaced with adjusted values from nearby stations used as “backup”. The Illinois Climate Network (Water and Atmospheric Resources Monitoring Program, 2015) station at St. Charles, Illinois is used as "backup" for the air temperature, solar radiation and wind speed data. Midwestern Regional Climate Center (Midwestern Regional Climate Center, 2017) station at Chicago O'Hare International Airport is used as "backup" for the dewpoint temperature and wind speed data. Each data source flag is of the form "xyz" that allows the user to determine its source and the methods used to process the data (Over and others, 2010). To open this file user needs to install any of the utilities described in the section "Related External Resources" in this page. References Cited: Argonne National Laboratory, 2017, Meteorological data, accessed on October 25, 2017, at URL http://gonzalo.er.anl.gov/ANLMET/. Midwestern Regional Climate Center, 2017, Meteorological data, accessed on December 5, 2017, at URL http://mrcc.isws.illinois.edu/CLIMATE/welcome.jsp. Murphy, E.A., 2005, Comparison of potential evapotranspiration calculated by the LXPET (Lamoreux Potential Evapotranspiration) Program and by the WDMUtil (Watershed Data Management Utility) Program: U.S. Geological Survey Open-File Report 2005-1020, 20 p., https://pubs.er.usgs.gov/publication/ofr20051020. Over, T.M., Price, T.H., and Ishii, A.L., 2010, Development and analysis of a meteorological database, Argonne National Laboratory, Illinois: U.S. Geological Survey Open-File Report 2010-1220, 67 p., http://pubs.usgs.gov/of/2010/1220/. Water and Atmospheric Resources Monitoring Program, 2015, Illinois Climate Network: Champaign, Ill., Illinois State Water Survey, accessed on December 5, 2017, at http://dx.doi.org/10.13012/J8MW2F2Q.

This is a dataset download, not a document. The Open button will start the download.This data layer is an element of the Oregon GIS Framework. The Oregon Biodiversity Information Center (ORBIC), part of the Institute for Natural Resources (INR) within the Oregon University System, has been the steward of Oregon’s protected areas data since 1989. This data is incorporated into the NavigatOR GIS utility and the national US protected areas database maintained by the U.S. Geological Survey. New data in Oregon on conservation easements and newly developed protected area maps from local land trusts and County and City governments were incorporated in 2011-2013. The result is a very comprehensive map and protected areas database for Oregon. Updates and edits will continue to be made to improve the dataset.

Zeolite-templated carbon (ZTC) is a unique porous carbonaceous material in that its structure is ordered at the nanometre scale, enabling a representative periodic description at the atomistic level. A structural library for ZTC of varying compositions was created using density functional tight binding (DFTB) potentials parameterized for materials science applications (matsci-0-3). We provide here quantum chemical-refined structures of models with CH, CHO, CHON, CHOB, and CHOBN compositions with various degrees of heteroatom substitution. The "initial ZTC structure" files correspond to the initial model used in our work that was developed using molecular mechanics, empirical force fields. These structural models comprise the characteristic morphological features of highly porous carbon materials, such as open-blade surfaces, edges, saddles, and closed-strut formations, spanning a range of curvatures and characteristic sizes. The optimized structures in CIF and native DFTB file formats are organized in the "stationary structure" file based on the optimization pathways that lead to the stationary structures.

Secondly, we carried out alternating compression and expansion of the CHO model unit cell to determine the lowest energy structure as well as to obtain the bulk modulus. The file "bulk modulus" contains two data sets that describe the deformational energy landscape of pure faujasite zeolite, Na-substituted zeolite, and the ZTC model structure.

The file "analysis tools" is a representative compilation of utilities for file format conversion, fractional vs. Cartesian crystal coordinates, and structural analysis spreadsheets.

The agreement between experimental measurements and the computational model is remarkable that demonstrates the power of approximate density functional theory as a cost-effective computational tool with chemical accuracy for the investigation of structure/property relationships in real-world carbon-based solids.

Geographic Diversity in Public Code Contributions - Replication Package

This document describes how to replicate the findings of the paper: Davide Rossi and Stefano Zacchiroli, 2022, Geographic Diversity in Public Code Contributions - An Exploratory Large-Scale Study Over 50 Years. In 19th International Conference on Mining Software Repositories (MSR ’22), May 23-24, Pittsburgh, PA, USA. ACM, New York, NY, USA, 5 pages. https://doi.org/10.1145/3524842.3528471

This document comes with the software needed to mine and analyze the data presented in the paper.

Prerequisites

These instructions assume the use of the bash shell, the Python programming language, the PosgreSQL DBMS (version 11 or later), the zstd compression utility and various usual *nix shell utilities (cat, pv, …), all of which are available for multiple architectures and OSs. It is advisable to create a Python virtual environment and install the following PyPI packages:

click==8.0.4 cycler==0.11.0 fonttools==4.31.2 kiwisolver==1.4.0 matplotlib==3.5.1 numpy==1.22.3 packaging==21.3 pandas==1.4.1 patsy==0.5.2 Pillow==9.0.1 pyparsing==3.0.7 python-dateutil==2.8.2 pytz==2022.1 scipy==1.8.0 six==1.16.0 statsmodels==0.13.2

Initial data

swh-replica, a PostgreSQL database containing a copy of Software Heritage data. The schema for the database is available at https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/sql/. We retrieved these data from Software Heritage, in collaboration with the archive operators, taking an archive snapshot as of 2021-07-07. We cannot make these data available in full as part of the replication package due to both its volume and the presence in it of personal information such as user email addresses. However, equivalent data (stripped of email addresses) can be obtained from the Software Heritage archive dataset, as documented in the article: Antoine Pietri, Diomidis Spinellis, Stefano Zacchiroli, The Software Heritage Graph Dataset: Public software development under one roof. In proceedings of MSR 2019: The 16th International Conference on Mining Software Repositories, May 2019, Montreal, Canada. Pages 138-142, IEEE 2019. http://dx.doi.org/10.1109/MSR.2019.00030. Once retrieved, the data can be loaded in PostgreSQL to populate swh-replica.

names.tab - forenames and surnames per country with their frequency

zones.acc.tab - countries/territories, timezones, population and world zones

c_c.tab - ccTDL entities - world zones matches

Data preparation

Export data from the swh-replica database to create commits.csv.zst and authors.csv.zst

sh> ./export.sh

Run the authors cleanup script to create authors--clean.csv.zst

sh> ./cleanup.sh authors.csv.zst

Filter out implausible names and create authors--plausible.csv.zst

sh> pv authors--clean.csv.zst | unzstd | ./filter_names.py 2> authors--plausible.csv.log | zstdmt > authors--plausible.csv.zst

Zone detection by email

Run the email detection script to create author-country-by-email.tab.zst

sh> pv authors--plausible.csv.zst | zstdcat | ./guess_country_by_email.py -f 3 2> author-country-by-email.csv.log | zstdmt > author-country-by-email.tab.zst

Database creation and initial data ingestion

Create the PostgreSQL DB

sh> createdb zones-commit

Notice that from now on when prepending the psql> prompt we assume the execution of psql on the zones-commit database.

Import data into PostgreSQL DB

sh> ./import_data.sh

Zone detection by name

Extract commits data from the DB and create commits.tab, that is used as input for the zone detection script

sh> psql -f extract_commits.sql zones-commit

Run the world zone detection script to create commit_zones.tab.zst

sh> pv commits.tab | ./assign_world_zone.py -a -n names.tab -p zones.acc.tab -x -w 8 | zstdmt > commit_zones.tab.zst Use ./assign_world_zone.py --help if you are interested in changing the script parameters.

Ingest zones assignment data into the DB

psql> \copy commit_zone from program 'zstdcat commit_zones.tab.zst | cut -f1,6 | grep -Ev ''\s$'''

Extraction and graphs

Run the script to execute the queries to extract the data to plot from the DB. This creates commit_zones_7120.tab, author_zones_7120_t5.tab, commit_zones_7120.grid and author_zones_7120_t5.grid. Edit extract_data.sql if you whish to modify extraction parameters (start/end year, sampling, …).

sh> ./extract_data.sh

Run the script to create the graphs from all the previously extracted tabfiles.

sh> ./create_stackedbar_chart.py -w 20 -s 1971 -f commit_zones_7120.grid -f author_zones_7120_t5.grid -o chart.pdf

Delineation of power lines in Namibia, that share geographical overlapping with the Okavango Basin. Source: Ministry of Agriculture, Water and Forestry of Namibia. This dataset is part of the GIS Database for the Environment Protection and Sustainable Management of the Okavango River Basin project (EPSMO). Detailed information on the database can be found in the “GIS Database for the EPSMO Project†document produced by Luis Veríssimo (FAO consultant) in July 2009, and here available for download.

Watershed Data Management (WDM) database file ARGN20.WDM is an update of ARGN19.WDM (Bera, 2020) with the processed data from October 1, 2019 through September 30, 2020, appended to it. The primary data were downloaded from the Argonne National Laboratory (ANL) (Argonne National Laboratory, 2020) and processed following the guidelines documented in Over and others (2010). ARGN20.WDM file contains nine data series: air temperature, in degrees Fahrenheit (dsn 400), dewpoint temperature, in degrees Fahrenheit (dsn 500), wind speed, in miles per hour (dsn 300), solar radiation, in Langleys (dsn 600), computed potential evapotranspiration, in thousandths of an inch (dsn 200), and four data-source flag series for air temperature (dsn 410), dewpoint temperature (dsn 510), wind speed (dsn 310), and solar radiation (dsn 610), respectively, from January 1,1948, to September 30, 2020. Daily potential evapotranspiration (PET) were computed from average daily air temperature, average daily dewpoint temperature, daily total wind speed, and daily total solar radiation and disaggregated to hourly PET, in thousandths of an inch, using the Fortran program LXPET (Murphy, 2005). Missing and apparently erroneous data values were replaced with adjusted values from nearby weather stations used as “backup”. The Illinois Climate Network (Water and Atmospheric Resources Monitoring Program, 2020) station at St. Charles, Illinois, was used as "backup" for the hourly air temperature, solar radiation, and wind speed data. The Midwestern Regional Climate Center (Midwestern Regional Climate Center, 2020) provided the hourly dewpoint temperature and wind speed data collected by the National Weather Service from the station at O'Hare International Airport and used as "backup". Each data source flag is of the form "xyz", which allows the user to determine its source and the methods used to process the data (Over and others, 2010). To open this file user needs to install any of the utilities described in the section "Related External Resources" on this page. References Cited: Argonne National Laboratory, 2020, Meteorological data, accessed on November 17, 2020, at http://www.atmos.anl.gov/ANLMET/. Bera, M., 2020, Meteorological Database, Argonne National Laboratory, Illinois, January 1, 1948 - September 30, 2019: U.S. Geological Survey data release, ​https://doi.org/10.5066/P9X0P4HZ. Midwestern Regional Climate Center, 2020, Meteorological data, accessed on November 3, 2020, at https://mrcc.illinois.edu/CLIMATE/. Murphy, E.A., 2005, Comparison of potential evapotranspiration calculated by the LXPET (Lamoreux Potential Evapotranspiration) Program and by the WDMUtil (Watershed Data Management Utility) Program: U.S. Geological Survey Open-File Report 2005-1020, 20 p., https://pubs.er.usgs.gov/publication/ofr20051020. Over, T.M., Price, T.H., and Ishii, A.L., 2010, Development and analysis of a meteorological database, Argonne National Laboratory, Illinois: U.S. Geological Survey Open-File Report 2010-1220, 67 p., http://pubs.usgs.gov/of/2010/1220/. Water and Atmospheric Resources Monitoring Program. Illinois Climate Network, 2020. Illinois State Water Survey, 2204 Griffith Drive, Champaign, IL 61820-7495. Data accessed on November 9, 2020, at http://dx.doi.org/10.13012/J8MW2F2Q.

Watershed Data Management (WDM) database file ARGN18.WDM is an update of ARGN17.WDM (Bera and Over, 2018) with the processed data from October 1, 2017 through September 30, 2018 appended to it. The primary data were downloaded from the Argonne National Laboratory (ANL) (Argonne National Laboratory, 2018) and processed following the guidelines documented in Over and others (2010). ARGN18.WDM file contains nine data series: air temperature, in degrees Fahrenheit (dsn 400), dewpoint temperature, in degrees Fahrenheit (dsn 500), wind speed, in miles per hour (dsn 300), solar radiation, in Langleys (dsn 600), computed potential evapotranspiration, in thousandths of an inch (dsn 200), and four data-source flag series for air temperature (dsn 410), dewpoint temperature (dsn 510), wind speed (dsn 310), and solar radiation (dsn 610), respectively, from January 1,1948, to September 30, 2018. Daily potential evapotranspiration (PET) were computed from average daily air temperature, average daily dewpoint temperature, daily total wind speed, and daily total solar radiation and disaggregated to hourly PET, in thousandths of an inch, using the Fortran program LXPET (Murphy, 2005). Missing and apparently erroneous data values were replaced with adjusted values from nearby weather stations used as “backup”. The Illinois Climate Network (Water and Atmospheric Resources Monitoring Program, 2018) station at St. Charles, Illinois, was used as "backup" for the hourly air temperature, solar radiation, and wind speed data. The Midwestern Regional Climate Center (Midwestern Regional Climate Center, 2018) provided the hourly dewpoint temperature and wind speed data collected by the National Weather Service at the station at O'Hare International Airport and used as "backup". Each data source flag is of the form "xyz", which allows the user to determine its source and the methods used to process the data (Over and others, 2010). To open this file user needs to install any of the utilities described in the section "Related External Resources" on this page. References Cited: Argonne National Laboratory, 2018, Meteorological data, accessed on October 10, 2018, at http://www.atmos.anl.gov/ANLMET/. Bera, M., and Over, T.M., 2018, Meteorological Database, Argonne National Laboratory, Illinois, January 1, 1948 - September 30, 2017: U.S. Geological Survey data release, ​https://doi.org/10.5066/F7H1318R. Midwestern Regional Climate Center, 2018, Meteorological data, accessed on October 12, 2018, at https://mrcc.illinois.edu/CLIMATE/. Murphy, E.A., 2005, Comparison of potential evapotranspiration calculated by the LXPET (Lamoreux Potential Evapotranspiration) Program and by the WDMUtil (Watershed Data Management Utility) Program: U.S. Geological Survey Open-File Report 2005-1020, 20 p., https://pubs.er.usgs.gov/publication/ofr20051020. Over, T.M., Price, T.H., and Ishii, A.L., 2010, Development and analysis of a meteorological database, Argonne National Laboratory, Illinois: U.S. Geological Survey Open-File Report 2010-1220, 67 p., http://pubs.usgs.gov/of/2010/1220/. Water and Atmospheric Resources Monitoring Program. Illinois Climate Network, 2018. Illinois State Water Survey, 2204 Griffith Drive, Champaign, IL 61820-7495. Data accessed on October 30, 2018, at http://dx.doi.org/10.13012/J8MW2F2Q

Special Database 19 contains NIST's entire corpus of training materials for handprinted document and character recognition. It publishes Handprinted Sample Forms from 3600 writers, 810,000 character images isolated from their forms, ground truth classifications for those images, reference forms for further data collection, and software utilities for image management and handling. there are two editions of the databases. One is the original database with the images in mis or pct format. It also includes software to open and manipulate the data. The second edition has the images all in PNG format.