Context

NYC Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker. Source: https://opendata.cityofnewyork.us/overview/

Content

Thanks to NYC Open Data, which makes public data generated by city agencies available for public use, and Citi Bike, we've incorporated over 150 GB of data in 5 open datasets into Google BigQuery Public Datasets, including:

• Over 8 million 311 service requests from 2012-2016

• More than 1 million motor vehicle collisions 2012-present

• Citi Bike stations and 30 million Citi Bike trips 2013-present

• Over 1 billion Yellow and Green Taxi rides from 2009-present

• Over 500,000 sidewalk trees surveyed decennially in 1995, 2005, and 2015

This dataset is deprecated and not being updated.

Fork this kernel to get started with this dataset.

Acknowledgements

https://opendata.cityofnewyork.us/

https://cloud.google.com/blog/big-data/2017/01/new-york-city-public-datasets-now-available-on-google-bigquery

This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - https://data.cityofnewyork.us/ - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

By accessing datasets and feeds available through NYC Open Data, the user agrees to all of the Terms of Use of NYC.gov as well as the Privacy Policy for NYC.gov. The user also agrees to any additional terms of use defined by the agencies, bureaus, and offices providing data. Public data sets made available on NYC Open Data are provided for informational purposes. The City does not warranty the completeness, accuracy, content, or fitness for any particular purpose or use of any public data set made available on NYC Open Data, nor are any such warranties to be implied or inferred with respect to the public data sets furnished therein.

The City is not liable for any deficiencies in the completeness, accuracy, content, or fitness for any particular purpose or use of any public data set, or application utilizing such data set, provided by any third party.

Banner Photo by @bicadmedia from Unplash.

Inspiration

On which New York City streets are you most likely to find a loud party?

Can you find the Virginia Pines in New York City?

Where was the only collision caused by an animal that injured a cyclist?

What’s the Citi Bike record for the Longest Distance in the Shortest Time (on a route with at least 100 rides)?

https://cloud.google.com/blog/big-data/2017/01/images/148467900588042/nyc-dataset-6.png" alt="enter image description here"> https://cloud.google.com/blog/big-data/2017/01/images/148467900588042/nyc-dataset-6.png

Temperature and precipitation projections for NYC reported by the New York City Panel on Climate Change (NPCC).

The New York City Panel on Climate Change (NPCC) started in 2009 and was codified in Local Law 42 of 2012 with a mandate to provide an authoritative and actionable source of scientific information on future climate change and its potential impacts.

The Intergovernmental Panel on Climate Change (IPCC) is the United Nations body for assessing the science related to climate change.

This dataset contains the list of dataset nominations submitted to the NYC Open Data team.

A single line street base map representing the city's streets and other linear geographic features, along with feature names and address ranges for each addressable street segment. This dataset includes the Nodes file. The Nodes file contains a point feature and unique NodeID for each node that exists in the LION file. The Node_StreetName.txt file lists the street names associated with those nodes. Most nodes, representing intersections, will have at least 2 street names associated in the Node_StreetName.txt file.

All previously released versions of this data are available on the DCP Website: BYTES of the BIG APPLE. Current version: 25c

gradio/NYC-Airbnb-Open-Data dataset hosted on Hugging Face and contributed by the HF Datasets community

This list contains information on approved event applications from 2008. Please note that Permitted Film Events only reflect those permits which will impact one or more streets for at least five days. For a current list of events, please refer to NYC Permitted Event Information dataset at https://data.cityofnewyork.us/City-Government/NYC-Permitted-Event-Information/tvpp-9vvx

This dataset compiles a comprehensive database containing 90,327 street segments in New York City, covering their street design features, streetscape design, Vision Zero treatments, and neighborhood land use. It has two scales-street and street segment group (aggregation of same type of street at neighborhood). This dataset is derived based on all publicly available data, most from NYC Open Data. The detailed methods can be found in the published paper, Pedestrian and Car Occupant Crash Casualties Over a 9-Year Span of Vision Zero in New York City. To use it, please refer to the metadata file for more information and cite our work. A full list of raw data source can be found below:

Motor Vehicle Collisions – NYC Open Data: https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95

Citywide Street Centerline (CSCL) – NYC Open Data: https://data.cityofnewyork.us/City-Government/NYC-Street-Centerline-CSCL-/exjm-f27b

NYC Building Footprints – NYC Open Data: https://data.cityofnewyork.us/Housing-Development/Building-Footprints/nqwf-w8eh

Practical Canopy for New York City: https://zenodo.org/record/6547492

New York City Bike Routes – NYC Open Data: https://data.cityofnewyork.us/Transportation/New-York-City-Bike-Routes/7vsa-caz7

Sidewalk Widths NYC (originally from Sidewalk – NYC Open Data): https://www.sidewalkwidths.nyc/

LION Single Line Street Base Map - The NYC Department of City Planning (DCP): https://www.nyc.gov/site/planning/data-maps/open-data/dwn-lion.page

NYC Planimetric Database Median – NYC Open Data: https://data.cityofnewyork.us/Transportation/NYC-Planimetrics/wt4d-p43d

NYC Vision Zero Open Data (including multiple datasets including all the implementations): https://www.nyc.gov/content/visionzero/pages/open-data

NYS Traffic Data - New York State Department of Transportation Open Data: https://data.ny.gov/Transportation/NYS-Traffic-Data-Viewer/7wmy-q6mb

Smart Location Database - US Environmental Protection Agency: https://www.epa.gov/smartgrowth/smart-location-mapping

Race and ethnicity in area - American Community Survey (ACS): https://www.census.gov/programs-surveys/acs

The Uber Ride Dataset for New York City contains detailed information about every Uber ride in the city. The dataset includes the TLC license number of the HVFHS base or business, the TLC Base License Number of the base that dispatched the trip, the date and time of the trip pick-up and drop-off, the TLC Taxi Zone in which the trip began and ended, the base number of the base that received the original trip request, and the date and time when the passenger requested to be picked up.

The dataset also provides information about the total miles for the passenger trip, the total time in seconds for the passenger trip, the base passenger fare before tolls, tips, taxes, and fees, the total amount of all tolls paid in the trip, the total amount collected in the trip for the Black Car Fund, the total amount collected in the trip for NYS sales tax, the total amount collected in the trip for NYS congestion surcharge, and the airport fee of $2.50 for both drop off and pick up at LaGuardia, Newark, and John F. Kennedy airports.

Moreover, the dataset includes the total amount of tips received from the passenger, the total driver pay (not including tolls or tips and net of commission, surcharges, or taxes), the flag indicating whether the passenger agreed to a shared/pooled ride and whether the passenger shared the vehicle with another passenger who booked separately at any point during the trip.

The dataset also includes information about whether the trip was administered on behalf of the Metropolitan Transportation Authority (MTA), whether the passenger requested a wheelchair-accessible vehicle (WAV), and whether the trip occurred in a wheelchair-accessible vehicle (WAV). This comprehensive dataset can be used for a variety of research and analysis purposes, including traffic patterns, fare analysis, and more.

The datasets are broken down by month and formatted in parquet. To use the parquet formatted files in pandas, there is an example in my notebook in the code section. If you need more details, look at the pdfs in the datasets. The data is originally from https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

The Department of State keeps a record of every filing for every incorporated business in the state of New York. This dataset contains information on all active corporations as of the last business day of the specified month and year.

Web traffic statistics for the top 2000 most visited pages on nyc.gov by month.

This dataset identifies property managed partially or solely by NYC Parks. This data has been produced in whole or part using secondary data. Data accuracy is limited by the scale and accuracy of the original sources. Site-specific conditions should be field-verified.

Records are added as more land is designated under NYC Parks’ jurisdiction. Each record represents an acquisition.

User Guide: https://docs.google.com/document/d/1NExNJF5YKID04oOopi0fHainRuGG3Pz_jKSrMujPsPk/edit?usp=sharing

Data Dictionary: https://docs.google.com/spreadsheets/d/1Q4DBWu7riNFxWvy1vnTJHoOI3r2L9oW6eCN56jCNyCw/edit?usp=sharing

This dataset provides benefit, program, and resource information for over 80 health and human services available to NYC residents in all eleven local law languages. The data is kept up-to-date, including the most recent applications, eligibility requirements, and application dates. Information in this dataset is used on ACCESS NYC, Generation NYC, and Growing Up NYC. Reach out to products@nycopportunity.nyc.gov if you have any questions about this dataset. This data makes it easier for NYC residents to discover and be aware of multiple benefits they may be eligible for. NYC Opportunity Product team works with 15+ government agencies to collect and update this data. Each record in the dataset represents a benefit or program. Blank fields are NULL values in this dataset. The data can be used to develop new websites or directory resources to help residents to discover benefits they need. For access to the multilingual version of this dataset, please follow this link: https://data.cityofnewyork.us/City-Government/Benefits-and-Programs-Multilingual-Dataset/yjpx-srhp

A listing of all retail food stores which are licensed by the Department of Agriculture and Markets.

A brief history of water consumption in the New York City Water Supply System (Based on New York City Census population)

Annual Average Daily Traffic (AADT) is an estimate of the average daily traffic along a defined segment of roadway. This value is calculated from short term counts taken along the same section which are then factored to produce the estimate of AADT. Because of this process, the most recent AADT for any given roadway will always be for the previous year. Data is available for all New York State Routes and roads that are part of the Federal Aid System.

A list of all datasets that were identified for publication on NYC Open Data and their current release status. For comprehensive information on each dataset currently on NYC Open Data, please refer to Local Law 251 of 2017: Published Data Asset Inventory.

The NYC Department of City Planning’s (DCP) Housing Database contains all NYC Department of Buildings (DOB) approved housing construction and demolition jobs filed or completed in NYC since January 1, 2010. It includes the three primary construction job types that add or remove residential units: new buildings, major alterations, and demolitions, and can be used to determine the change in legal housing units across time and space. Records in the Housing Database Project-Level Files are geocoded to the greatest level of precision possible, subject to numerous quality assurance and control checks, recoded for usability, and joined to other housing data sources relevant to city planners and analysts. Data are updated semiannually, at the end of the second and fourth quarters of each year. Please see DCP’s annual Housing Production Snapshot summarizing findings from the 21Q4 data release here. Additional Housing and Economic analyses are also available. The NYC Department of City Planning’s (DCP) Housing Database Unit Change Summary Files provide the net change in Class A housing units since 2010, and the count of units pending completion for commonly used political and statistical boundaries (Census Block, Census Tract, City Council district, Community District, Community District Tabulation Area (CDTA), Neighborhood Tabulation Area (NTA). These tables are aggregated from the DCP Housing Database Project-Level Files, which is derived from Department of Buildings (DOB) approved housing construction and demolition jobs filed or completed in NYC since January 1, 2010. Net housing unit change is calculated as the sum of all three construction job types that add or remove residential units: new buildings, major alterations, and demolitions. These files can be used to determine the change in legal housing units across time and space.

Bridge strike occurrences on New York City roadways that have low clearances.

All 311 Service Requests from 2010 to present. This information is automatically updated daily.

Click here to download data from 2011 - https://data.cityofnewyork.us/dataset/311-Service-Requests-From-2011/fpz8-jqf4

Click here to download data from 2012 - https://data.cityofnewyork.us/dataset/311-Service-Requests-From-2012/as38-8eb5

Click here to download data from 2013 - https://data.cityofnewyork.us/dataset/311-Service-Requests-From-2013/hybb-af8n

Click here to download data from 2014 - https://data.cityofnewyork.us/dataset/311-Service-Requests-From-2014/vtzg-7562

Click here to download data from 2015 - https://data.cityofnewyork.us/dataset/311-Service-Requests-From-2015/57g5-etyj

This dataset represents amenities activated as a part of Cool It! NYC, a Citywide plan to increase the amount of cooling features available to the public during heat emergencies, particularly in neighborhoods that face the dangers of high heat. This is part of the Cool It! NYC 2020 Data Collection, which includes the following amenities:

Drinking Fountains: Indicates whether a drinking fountain is activated, not yet activated, broken, or under construction.

Spray Showers: Indicates whether a spray shower installed before July 2020 is activated, not yet activated, broken, or under construction. At this time, spray showers are mapped to the middle of parks.

Cooling Sites: To measure neighborhoods that are the most at risk during extreme heat, NYC Health and Columbia University developed the New York City Heat Vulnerability Index, or HVI. Parks used this data to direct new cooling elements to neighborhoods with HVIs of 4 and 5.

Data Dictionary: https://docs.google.com/spreadsheets/d/1GpXHX9p0e520LcAf3gstOKTQm64wxkdDUiACjhMwd9Q/edit?usp=sharing