Context

NYC Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker. Source: https://opendata.cityofnewyork.us/overview/

Content

Thanks to NYC Open Data, which makes public data generated by city agencies available for public use, and Citi Bike, we've incorporated over 150 GB of data in 5 open datasets into Google BigQuery Public Datasets, including:

• Over 8 million 311 service requests from 2012-2016

• More than 1 million motor vehicle collisions 2012-present

• Citi Bike stations and 30 million Citi Bike trips 2013-present

• Over 1 billion Yellow and Green Taxi rides from 2009-present

• Over 500,000 sidewalk trees surveyed decennially in 1995, 2005, and 2015

This dataset is deprecated and not being updated.

Fork this kernel to get started with this dataset.

Acknowledgements

https://opendata.cityofnewyork.us/

https://cloud.google.com/blog/big-data/2017/01/new-york-city-public-datasets-now-available-on-google-bigquery

This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source - https://data.cityofnewyork.us/ - and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

By accessing datasets and feeds available through NYC Open Data, the user agrees to all of the Terms of Use of NYC.gov as well as the Privacy Policy for NYC.gov. The user also agrees to any additional terms of use defined by the agencies, bureaus, and offices providing data. Public data sets made available on NYC Open Data are provided for informational purposes. The City does not warranty the completeness, accuracy, content, or fitness for any particular purpose or use of any public data set made available on NYC Open Data, nor are any such warranties to be implied or inferred with respect to the public data sets furnished therein.

The City is not liable for any deficiencies in the completeness, accuracy, content, or fitness for any particular purpose or use of any public data set, or application utilizing such data set, provided by any third party.

Banner Photo by @bicadmedia from Unplash.

Inspiration

On which New York City streets are you most likely to find a loud party?

Can you find the Virginia Pines in New York City?

Where was the only collision caused by an animal that injured a cyclist?

What’s the Citi Bike record for the Longest Distance in the Shortest Time (on a route with at least 100 rides)?

https://cloud.google.com/blog/big-data/2017/01/images/148467900588042/nyc-dataset-6.png" alt="enter image description here"> https://cloud.google.com/blog/big-data/2017/01/images/148467900588042/nyc-dataset-6.png

The NYC Department of City Planning’s (DCP) Housing Database contains all NYC Department of Buildings (DOB) approved housing construction and demolition jobs filed or completed in NYC since January 1, 2010. It includes the three primary construction job types that add or remove residential units: new buildings, major alterations, and demolitions, and can be used to determine the change in legal housing units across time and space. Records in the Housing Database Project-Level Files are geocoded to the greatest level of precision possible, subject to numerous quality assurance and control checks, recoded for usability, and joined to other housing data sources relevant to city planners and analysts. Data are updated semiannually, at the end of the second and fourth quarters of each year. Please see DCP’s annual Housing Production Snapshot summarizing findings from the 21Q4 data release here. Additional Housing and Economic analyses are also available. The NYC Department of City Planning’s (DCP) Housing Database Unit Change Summary Files provide the net change in Class A housing units since 2010, and the count of units pending completion for commonly used political and statistical boundaries (Census Block, Census Tract, City Council district, Community District, Community District Tabulation Area (CDTA), Neighborhood Tabulation Area (NTA). These tables are aggregated from the DCP Housing Database Project-Level Files, which is derived from Department of Buildings (DOB) approved housing construction and demolition jobs filed or completed in NYC since January 1, 2010. Net housing unit change is calculated as the sum of all three construction job types that add or remove residential units: new buildings, major alterations, and demolitions. These files can be used to determine the change in legal housing units across time and space.

Web traffic statistics for the top 2000 most visited pages on nyc.gov by month.

The leading causes of death by sex and ethnicity in New York City in since 2007. Cause of death is derived from the NYC death certificate which is issued for every death that occurs in New York City. Report last ran: 09/24/2019 Rates based on small numbers (RSE > 30) as well as aggregate counts less than 5 have been suppressed in downloaded data Source: Bureau of Vital Statistics and New York City Department of Health and Mental Hygiene

In 2024, the City of New York experienced a total of ******* felonies. This was a large decrease from 2001 when ******* felonies were reported. These figures comprise the seven major categories of felonies that are listed by the New York Police Department (NYPD) for statistical analysis. They are murder and non-negligible manslaughter, rape, robbery, felony assault, burglary, grand larceny, and grand larceny of motor vehicle.

Data that that populates the Vision Zero View map, which can be found at www.nycvzv.info Vision Zero is the City's goal for ending traffic deaths and injuries. The Vision Zero action plan can be found at http://www.nyc.gov/html/visionzero/pdf/nyc-vision-zero-action-plan.pdf Crash data is obtained from the Traffic Accident Management System (TAMS), which is maintained by the New York City Police Department (NYPD). Only crashes with valid geographic information are mapped. All midblock crashes are mapped to the nearest intersection. Injuries and fatalities are grouped by intersection and summarized by month and year. This data is queried and aggregated on a monthly basis and is current as of the query date. Current year data is January to the end of the latest full month. All mappable crash data is represented on the simplified NYC street model. Crashes occurring at complex intersections with multiple roadways are mapped onto a single point. Injury and fatality crashes occurring on highways are excluded from this data. Please note that this data is preliminary and may contain errors, accordingly, the data on this site is for informational purposes only. Although all attempts to provide the most accurate information are made, errors may be present and any person who relies upon this data does so at their own risk.

A 2024 study reported that New York City, United States, received just over ** million visitors in 2023. The previous year, the city saw **** million visitors. Projections suggest that this number could rise to as many as ** million by the year 2025.

A list of all datasets that were identified for publication on NYC Open Data and their current release status. For comprehensive information on each dataset currently on NYC Open Data, please refer to Local Law 251 of 2017: Published Data Asset Inventory.

Bridge strike occurrences on New York City roadways that have low clearances.

Note: As of November 10, 2023, this dataset has been archived. For the current version of this data, please visit: https://health.data.ny.gov/d/gikn-znjh

This dataset reports daily on the number of people vaccinated by New York providers with at least one dose and with a complete COVID-19 vaccination series overall since December 14, 2020. New York providers include hospitals, mass vaccination sites operated by the State or local governments, pharmacies, and other providers registered with the State to serve as points of distribution.

This dataset is created by the New York State Department of Health from data reported to the New York State Immunization Information System (NYSIIS) and the New York City Citywide Immunization Registry (NYC CIR). County-level vaccination data is based on data reported to NYSIIS and NYC CIR by the providers administering vaccines. Residency is self-reported by the individual being vaccinated. This data does not include vaccine administered through Federal entities or performed outside of New York State to New York residents. NYSIIS and CIR data is used for county-level statistics. New York State Department of Health requires all New York State vaccination providers to report all COVID-19 vaccination administration data to NYSIIS and NYC CIR within 24 hours of administration.

This dataset contains building information for all buildings that have completed a WiredNYC survey. This includes buildings that have opted-out from displaying their profiles publicly. Therefore, the building-specific data (e.g. building address) provided is anonymous and only linked to the borough the building is located in.

Annual Average Daily Traffic (AADT) is an estimate of the average daily traffic along a defined segment of roadway. This value is calculated from short term counts taken along the same section which are then factored to produce the estimate of AADT. Because of this process, the most recent AADT for any given roadway will always be for the previous year. Data is available for all New York State Routes and roads that are part of the Federal Aid System.

This dataset compiles a comprehensive database containing 90,327 street segments in New York City, covering their street design features, streetscape design, Vision Zero treatments, and neighborhood land use. It has two scales-street and street segment group (aggregation of same type of street at neighborhood). This dataset is derived based on all publicly available data, most from NYC Open Data. The detailed methods can be found in the published paper, Pedestrian and Car Occupant Crash Casualties Over a 9-Year Span of Vision Zero in New York City. To use it, please refer to the metadata file for more information and cite our work. A full list of raw data source can be found below:

Motor Vehicle Collisions – NYC Open Data: https://data.cityofnewyork.us/Public-Safety/Motor-Vehicle-Collisions-Crashes/h9gi-nx95

Citywide Street Centerline (CSCL) – NYC Open Data: https://data.cityofnewyork.us/City-Government/NYC-Street-Centerline-CSCL-/exjm-f27b

NYC Building Footprints – NYC Open Data: https://data.cityofnewyork.us/Housing-Development/Building-Footprints/nqwf-w8eh

Practical Canopy for New York City: https://zenodo.org/record/6547492

New York City Bike Routes – NYC Open Data: https://data.cityofnewyork.us/Transportation/New-York-City-Bike-Routes/7vsa-caz7

Sidewalk Widths NYC (originally from Sidewalk – NYC Open Data): https://www.sidewalkwidths.nyc/

LION Single Line Street Base Map - The NYC Department of City Planning (DCP): https://www.nyc.gov/site/planning/data-maps/open-data/dwn-lion.page

NYC Planimetric Database Median – NYC Open Data: https://data.cityofnewyork.us/Transportation/NYC-Planimetrics/wt4d-p43d

NYC Vision Zero Open Data (including multiple datasets including all the implementations): https://www.nyc.gov/content/visionzero/pages/open-data

NYS Traffic Data - New York State Department of Transportation Open Data: https://data.ny.gov/Transportation/NYS-Traffic-Data-Viewer/7wmy-q6mb

Smart Location Database - US Environmental Protection Agency: https://www.epa.gov/smartgrowth/smart-location-mapping

Race and ethnicity in area - American Community Survey (ACS): https://www.census.gov/programs-surveys/acs

The Uber Ride Dataset for New York City contains detailed information about every Uber ride in the city. The dataset includes the TLC license number of the HVFHS base or business, the TLC Base License Number of the base that dispatched the trip, the date and time of the trip pick-up and drop-off, the TLC Taxi Zone in which the trip began and ended, the base number of the base that received the original trip request, and the date and time when the passenger requested to be picked up.

The dataset also provides information about the total miles for the passenger trip, the total time in seconds for the passenger trip, the base passenger fare before tolls, tips, taxes, and fees, the total amount of all tolls paid in the trip, the total amount collected in the trip for the Black Car Fund, the total amount collected in the trip for NYS sales tax, the total amount collected in the trip for NYS congestion surcharge, and the airport fee of $2.50 for both drop off and pick up at LaGuardia, Newark, and John F. Kennedy airports.

Moreover, the dataset includes the total amount of tips received from the passenger, the total driver pay (not including tolls or tips and net of commission, surcharges, or taxes), the flag indicating whether the passenger agreed to a shared/pooled ride and whether the passenger shared the vehicle with another passenger who booked separately at any point during the trip.

The dataset also includes information about whether the trip was administered on behalf of the Metropolitan Transportation Authority (MTA), whether the passenger requested a wheelchair-accessible vehicle (WAV), and whether the trip occurred in a wheelchair-accessible vehicle (WAV). This comprehensive dataset can be used for a variety of research and analysis purposes, including traffic patterns, fare analysis, and more.

The datasets are broken down by month and formatted in parquet. To use the parquet formatted files in pandas, there is an example in my notebook in the code section. If you need more details, look at the pdfs in the datasets. The data is originally from https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

The Department of Housing Preservation and Development (HPD) records complaints that are made by the public for conditions which violate the New York City Housing Maintenance Code (HMC) or the New York State Multiple Dwelling Law (MDL).

This dataset contains the collection and maintenance of crime data for incidents that occur in New York City public schools.

Many residents of New York City speak more than one language; a number of them speak and understand non-English languages more fluently than English. This dataset, derived from the Census Bureau's American Community Survey (ACS), includes information on over 1.7 million limited English proficient (LEP) residents and a subset of that population called limited English proficient citizens of voting age (CVALEP) at the Community District level. There are 59 community districts throughout NYC, with each district being represented by a Community Board.

This dataset provides benefit, program, and resource information for over 80 health and human services available to NYC residents in all eleven local law languages. The data is kept up-to-date, including the most recent applications, eligibility requirements, and application dates. Information in this dataset is used on ACCESS NYC, Generation NYC, and Growing Up NYC. Reach out to products@nycopportunity.nyc.gov if you have any questions about this dataset. This data makes it easier for NYC residents to discover and be aware of multiple benefits they may be eligible for. NYC Opportunity Product team works with 15+ government agencies to collect and update this data. Each record in the dataset represents a benefit or program. Blank fields are NULL values in this dataset. The data can be used to develop new websites or directory resources to help residents to discover benefits they need. For access to the multilingual version of this dataset, please follow this link: https://data.cityofnewyork.us/City-Government/Benefits-and-Programs-Multilingual-Dataset/yjpx-srhp

Data extracted from records of tickets on file with NYS DMV. The tickets were issued to motorists for violations of: NYS Vehicle & Traffic Law (VTL), Thruway Rules and Regulations, Tax Law, Transportation Law, Parks and Recreation Regulations, Local New York City Traffic Ordinances, and NYS Penal Law pertaining to the involvement of a motor vehicle in acts of assault, homicide, manslaughter and criminal negligence resulting in injury or death.

The Division of Criminal Justice Services (DCJS) collects crime reports from more than 500 New York State police and sheriffs’ departments. DCJS compiles these reports as New York’s official crime statistics and submits them to the FBI under the National Uniform Crime Reporting (UCR) Program. UCR uses standard offense definitions to count crime in localities across America regardless of variations in crime laws from state to state. In New York State, law enforcement agencies use the UCR system to report their monthly crime totals to DCJS. The UCR reporting system collects information on seven crimes classified as Index offenses which are most commonly used to gauge overall crime volume. These include the violent crimes of murder/non-negligent manslaughter, forcible rape, robbery, and aggravated assault; and the property crimes of burglary, larceny, and motor vehicle theft. Police agencies may experience reporting problems that preclude accurate or complete reporting. The counts represent only crimes reported to the police but not total crimes that occurred.