A list of all datasets that were identified for publication on NYC Open Data and their current release status. For comprehensive information on each dataset currently on NYC Open Data, please refer to Local Law 251 of 2017: Published Data Asset Inventory.

This dataset contains the list of dataset nominations submitted to the NYC Open Data team.

This is the Open Data Program Overview.

Context

NYC Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker. Source: https://opendata.cityofnewyork.us/overview/

Content

Thanks to NYC Open Data, which makes public data generated by city agencies available for public use, and Citi Bike, we've incorporated over 150 GB of data in 5 open datasets into Google BigQuery Public Datasets, including:

• Over 8 million 311 service requests from 2012-2016

• More than 1 million motor vehicle collisions 2012-present

• Citi Bike stations and 30 million Citi Bike trips 2013-present

• Over 1 billion Yellow and Green Taxi rides from 2009-present

• Over 500,000 sidewalk trees surveyed decennially in 1995, 2005, and 2015

This dataset is deprecated and not being updated.

Fork this kernel to get started with this dataset.

Acknowledgements

https://opendata.cityofnewyork.us/

https://cloud.google.com/blog/big-data/2017/01/new-york-city-public-datasets-now-available-on-google-bigquery

This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - https://data.cityofnewyork.us/ - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

By accessing datasets and feeds available through NYC Open Data, the user agrees to all of the Terms of Use of NYC.gov as well as the Privacy Policy for NYC.gov. The user also agrees to any additional terms of use defined by the agencies, bureaus, and offices providing data. Public data sets made available on NYC Open Data are provided for informational purposes. The City does not warranty the completeness, accuracy, content, or fitness for any particular purpose or use of any public data set made available on NYC Open Data, nor are any such warranties to be implied or inferred with respect to the public data sets furnished therein.

The City is not liable for any deficiencies in the completeness, accuracy, content, or fitness for any particular purpose or use of any public data set, or application utilizing such data set, provided by any third party.

Banner Photo by @bicadmedia from Unplash.

Inspiration

On which New York City streets are you most likely to find a loud party?

Can you find the Virginia Pines in New York City?

Where was the only collision caused by an animal that injured a cyclist?

What’s the Citi Bike record for the Longest Distance in the Shortest Time (on a route with at least 100 rides)?

https://cloud.google.com/blog/big-data/2017/01/images/148467900588042/nyc-dataset-6.png" alt="enter image description here"> https://cloud.google.com/blog/big-data/2017/01/images/148467900588042/nyc-dataset-6.png

NOTE: To review the latest plan, make sure to filter the "Report Year" column to the latest year. Data on public websites maintained by or on behalf of the city agencies.

List of public APIs created using API Foundry that access Open.ny.gov datasets

Discover the breadth of data collected by the state which is local in nature. Search by county and municipality and discover, explore, and download local data. With a click, find local data across a broad range of categories from health to transportation, from recreation to economic development; find local farmer’s markets, child care regulated facilities, craft beverages, solar installations, food service establishment inspections, and much more.

NOTE: To review the latest plan, make sure to filter the "Report Year" column to the latest year. The list of datasets identified via FOIL reporting and their respective release statuses as a part of the Open Data Plan.

A single line street base map representing the city's streets and other linear geographic features, along with feature names and address ranges for each addressable street segment. This dataset includes the Nodes file. The Nodes file contains a point feature and unique NodeID for each node that exists in the LION file. The Node_StreetName.txt file lists the street names associated with those nodes. Most nodes, representing intersections, will have at least 2 street names associated in the Node_StreetName.txt file.

All previously released versions of this data are available on the DCP Website: BYTES of the BIG APPLE.

OPT provides transportation service to many different kinds of locations. Many of these locations are schools but they also include offices or other sites that may be part of certain students’ educational plans. The schools may be public, private or religious. OPT provides busing to some Pre-K sites for students who have an IEP for curb-to-curb busing because of medical condition. Transportation service is not limited to school bus service; it includes distribution of MetroCards and approved reimbursement services. Bus service can be conducted on a yellow school bus, an ambulance, or even a coach bus. Yellow school buses are available in a number of sizes and seating configurations. This dataset includes schools, offices or Pre-K/EI sites that currently receive any transportation services from OPT. These sites may be within the New York City limits or up to fifty miles from the city limits in the states of New York, New Jersey or Connecticut. This dataset does not include field trip destinations.

The Digital City Map (DCM) data represents street lines and other features shown on the City Map, which is the official street map of the City of New York. The City Map consists of 5 different sets of maps, one for each borough, totaling over 8000 individual paper maps. The DCM datasets were created in an ongoing effort to digitize official street records and bring them together with other street information to make them easily accessible to the public. The Digital City Map (DCM) is comprised of seven datasets; Digital City Map, Street Center Line, City Map Alterations, Arterial Highways and Major Streets, Street Name Changes (areas), Street Name Changes (lines), and Street Name Changes (points). All of the Digital City Map (DCM) datasets are featured on the Streets App All previously released versions of this data are available at BYTES of the BIG APPLE- Archive Updates for this dataset, along with other multilayered maps on NYC Open Data, are temporarily paused while they are moved to a new mapping format. Please visit https://www.nyc.gov/site/planning/data-maps/open-data/dwn-digital-city-map.page to utilize this data in the meantime.

A list of datasets that MTA currently shares and plans to share on data.ny.gov.

New York State's focus on data quality has been a hallmark of Open NY. This document is intended to be read together with the NYS Open Data Handbook, (https://data.ny.gov/dataset/NYS-Open-Data-Handbook/id8k-natf), and includes best practices garnered from lessons learned regarding optimal formatting and documentation. This guide represents a commitment to continuous quality improvement to maximize understanding, and the advancement of standardization to promote interoperability, analysis, and utilization of the data.

Open Government Data (OGD) has the potential to support social and economic progress. However, this potential can be frustrated if this data remains unused. Although the literature suggests that OGD datasets' metadata quality is one of the main factors affecting their use, to the best of our knowledge, no quantitative study provided evidence of this relationship. Considering about 400,000 datasets of 28 national, municipal, and international OGD portals, we have programmatically analyzed their usage, their metadata quality, and the relationship between the two. Our analysis has highlighted three main findings. First of all, regardless of their size, the software platform adopted, and their administrative and territorial coverage, most OGD datasets are underutilized. Second, OGD portals pay varying attention to the quality of their datasets’ metadata. Third, we did not find clear evidence that datasets usage is positively correlated to better metadata publishing practices. Finally, we have considered other factors, such as datasets’ category, and some demographic characteristics of the OGD portals, and analyzed their relationship with datasets usage, obtaining partially affirmative answers.

The dataset consists of three zipped CSV files, containing the collected datasets' usage data, full metadata, and computed quality values, for about 400,000 datasets belonging to the 8 national, 4 international, and 16 US municipalities OGD portals considered in the study.

Data collection occurred in the period: 2019-12-19 -- 2019-12-23.

Portal #Datasets Platform

US 261,514 CKAN

France 39,412 Other

Colombia 9,795 Socrata

IE 9,598 CKAN

Slovenia 4,892 CKAN

Poland 1,032 Other

Latvia 336 CKAN

Puerto Rico 178 Socrata

New York, NY 2,771 Socrata

Baltimore, MD 2,617 Socrata

Austin, TX 2,353 Socrata

Chicago, IL 1,368 Socrata

San Francisco, CA 1,001 Socrata

Dallas, TX 1,001 Socrata

Los Angeles, CA 943 Socrata

Seattle, WA 718 Socrata

Providence, RI 288 Socrata

Honolulu, HI 244 Socrata

New Orleans, LA 215 Socrata

Buffalo, NY 213 Socrata

Nashville, TN 172 Socrata

Boston, MA 170 CKAN

Albuquerque, NM 60 CKAN

Albany, NY 50 Socrata

HDX 17,325 CKAN

EUODP 14,058 CKAN

NASA 9,664 Socrata

World Bank Finances 2,177 Socrata

The three datasets share the same table structure:

Table Fields

portalid: portal identifier

id: dataset identifier

engine: identifier of the supporting portal platform: 1(CKAN), 2 (Socrata)

admindomain: 1 (National), 2 (US), 3 (International)

downloaddate: date of data collection

views: number of total views for the dataset

downloads: number of total downloads for the dataset

overallq: overall quality values computed by applying the methodology presented by Neumaier et al. in [1]

qvalues: json object containing the quality values computed for the 17 metrics presented in by Neumaier et al. [1]

assessdate: date of quality assessment

metadata: the overall dataset's metadata downloaded via API from the portal according to the supporting platform schema

[1] Neumaier, S.; Umbrich, J.; Polleres, A. Automated Quality Assessment of Metadata Across Open Data Portals.J. Data and Information Quality2016,8, 2:1–2:29. doi:10.1145/2964909

The Green Book Online is a fully searchable database which gives New Yorkers the opportunity to search for the agencies, offices, boards and commissions that keep our City running. It includes listings for all levels of New York City government, County, and the unified Courts system.

The Capital Projects Database reports information at the project level on discrete capital investments from the Capital Commitment Plan.Each row is uniquely identified by its Financial Management Service (FMS) ID, and contains data pertaining to the sponsoring and managing agency.

To explore the data, please visit Capital Planning Explorer

For additional information, please visit A Guide to The Capital Budget

This dataset provides the most current listing of LinkNYC Kiosks, their location, and the status of the Link’s wifi, tablet, and phone.

This dataset compiles a comprehensive database containing 90,327 street segments in New York City, covering their street design features, streetscape design, Vision Zero treatments, and neighborhood land use. It has two scales-street and street segment group (aggregation of same type of street at neighborhood). This dataset is derived based on all publicly available data, most from NYC Open Data. The detailed methods can be found in the published paper, Pedestrian and Car Occupant Crash Casualties Over a 9-Year Span of Vision Zero in New York City. To use it, please refer to the metadata file for more information and cite our work. A full list of raw data source can be found below:

Motor Vehicle Collisions – NYC Open Data: https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95

Citywide Street Centerline (CSCL) – NYC Open Data: https://data.cityofnewyork.us/City-Government/NYC-Street-Centerline-CSCL-/exjm-f27b

NYC Building Footprints – NYC Open Data: https://data.cityofnewyork.us/Housing-Development/Building-Footprints/nqwf-w8eh

Practical Canopy for New York City: https://zenodo.org/record/6547492

New York City Bike Routes – NYC Open Data: https://data.cityofnewyork.us/Transportation/New-York-City-Bike-Routes/7vsa-caz7

Sidewalk Widths NYC (originally from Sidewalk – NYC Open Data): https://www.sidewalkwidths.nyc/

LION Single Line Street Base Map - The NYC Department of City Planning (DCP): https://www.nyc.gov/site/planning/data-maps/open-data/dwn-lion.page

NYC Planimetric Database Median – NYC Open Data: https://data.cityofnewyork.us/Transportation/NYC-Planimetrics/wt4d-p43d

NYC Vision Zero Open Data (including multiple datasets including all the implementations): https://www.nyc.gov/content/visionzero/pages/open-data

NYS Traffic Data - New York State Department of Transportation Open Data: https://data.ny.gov/Transportation/NYS-Traffic-Data-Viewer/7wmy-q6mb

Smart Location Database - US Environmental Protection Agency: https://www.epa.gov/smartgrowth/smart-location-mapping

Race and ethnicity in area - American Community Survey (ACS): https://www.census.gov/programs-surveys/acs

This dataset is the full listing of Open Data NY Report download links.

A schedule of datasets that New York City agencies will make available on nyc.gov/data