This public dataset was created by the National Oceanic and Atmospheric Administration (NOAA) and includes global data obtained from the USAF Climatology Center. This dataset covers GSOD data between 1929 and present (updated daily), collected from over 9000 stations. Global summary of the day is comprised of a dozen daily averages computed from global hourly station data. Daily weather elements include mean values of: temperature, dew point temperature, sea level pressure, station pressure, visibility, and wind speed plus maximum and minimum temperature, maximum sustained wind speed and maximum gust, precipitation amount, snow depth, and weather indicators. With the exception of U.S. stations, 24-hour periods are based upon UTC times. This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery .

Context

Weather is the state of the atmosphere, describing for example the degree to which it is hot or cold, wet or dry, calm or stormy, clear or cloudy. Source: https://en.wikipedia.org/wiki/Weather

Content

NOAA’s Global Historical Climatology Network (GHCN) is an integrated database of climate summaries from land surface stations across the globe that have been subjected to a common suite of quality assurance reviews. Two GHCN datasets are available in BigQuery, the GHCN-D (daily) and the GHCN-M (monthly). The data included in the GHCN datasets are obtained from more than 20 sources, including some data from every year since 1763.

For a complete description of data variables available in this dataset, see NOAA’s readme.txt: https://www1.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt

Update Frequency: daily

Fork this kernel to get started with this dataset.

Acknowledgements

https://bigquery.cloud.google.com/dataset/bigquery-public-data:ghcn_d

https://cloud.google.com/bigquery/public-data/noaa-ghcn

Dataset Source: NOAA. This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source — http://www.data.gov/privacy-policy#data_policy — and is provided "AS IS" without any warranty, express or implied, from Google. Google disclaims all liability for any damages, direct or indirect, resulting from the use of the dataset.

Banner Photo by Max LaRochelle from Unplash.

Inspiration

Find weather stations close to a specific location?

Daily rainfall amounts at specific station?

Pulling daily min/max temperature (in Celsius) and rainfall (in mm) for the past 14 days?

NEXRAD Level 3 products are used to remotely detect atmospheric features, such as precipitation, precipitation-type, storms, turbulence and wind, for operational forecasting and data research analysis. Level 3 data consists of over 40 products that are the output product data of the Radar Product Generator. The products assist forecasters and others in weather analysis, forecasts, warnings, and weather tracking. The offerings from the Google Cloud Public Datasets Program include: Real-time NEXRAD L3 data Historic NEXRAD L3 data back to 1992 The Level 3 data consists of reduced resolution, low-bandwidth, base products and many derived, post-processed products. Level 3 products are recorded at most U.S. sites, though non-US sites do not have Level 3 products. General products for Level 3 include the base and composite reflectivity, storm relative velocity, vertical integrated liquid, echo tops and VAD wind profile. Precipitation products for Level 3 include estimated ground accumulated rainfall amounts for one and three hour periods, storm totals, and digital arrays. Estimates are based on reflectivity to rainfall rate (Z-R) relationships. Overlay products for Level 3 are alphanumeric data that give detailed information on certain parameters for an identified storm cell. These include storm structure, hail index, mesocyclone identification, tornadic vortex signature, and storm tracking information. You can see the current status of each radar site on NOAA's status site . This public dataset is hosted in Google Cloud Storage and available free to use. Use this quick start guide to quickly learn how to access public datasets on Google Cloud Storage.

The Alaska Fisheries Science Center is the research branch of the National Oceanic and Atmospheric Administration's National Marine Fisheries Service responsible for research on living marine resources in the coastal oceans off Alaska and off parts of the west coast of the United States. This region of nearly 3 million square miles includes the North Pacific Ocean and the eastern Bering Sea which support some of the most important commercial fisheries in the world. These waters are also home to the largest marine mammal populations in the Nation. The mission of the Alaska Fisheries Science Center is to plan, develop, and manage scientific research programs which generate the best scientific data available for understanding, managing, and conserving the region's living marine resources and the environmental quality essential for their existence. The Open Data Portal provides data and results from across the scope of the research conducted at the Alaska Fisheries Science Center. The data in this dataset have various expected update frequencies. Much of it is historic, while others are updated regularly. This public dataset is hosted in Google Cloud Storage and available free to use. Use this quick start guide to quickly learn how to access public datasets on Google Cloud Storage.

Overview

The Geostationary Operational Environmental Satellite-R Series (GOES-R) is the next generation of geostationary weather satellites. The GOES-R series will significantly improve the detection and observation of environmental phenomena that directly affect public safety, protection of property and our nation’s economic health and prosperity.

The GOES-16 satellite, known as GOES-R prior to launch, is the first satellite in the series. It will provide images of weather pattern and severe storms as frequently as every 30 seconds, which will contribute to more accurate and reliable weather forecasts and severe weather outlooks.

Content

The raw dataset includes a feed of the Advanced Baseline Imager (ABI) radiance data (Level 1b) and Cloud and Moisture Imager (CMI) products (Level 2) which are freely available through the NOAA Big Data Project.

Querying BigQuery tables

You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.github_repos.[TABLENAME]. Fork this kernel to get started to learn how to safely manage analyzing large BigQuery datasets.

Acknowledgments

The NOAA Big Data Project (BDP) is an experimental collaboration between NOAA and infrastructure-as-a-service (IaaS) providers to explore methods of expand the accessibility of NOAA’s data in order to facilitate innovation and collaboration. The goal of this approach is to help form new lines of business and economic growth while making NOAA's data more discoverable for the American public. https://storage.googleapis.com/public-dataset-images/noaa-goes-16-sample.png" alt="Sample images">

Key metadata for this dataset has been extracted into convenient BigQuery tables (one each for L1b radiance, L2 CMIP, and L2 MCMIP). These tables can be used to query metadata in order to filter the data down to only a subset of raw netcdf4 files available in Google Cloud Storage.

The Global Ensemble Forecast System (GEFS) has been operational at NCEP since December 1992, with the initial version using the NCEP Global Spectral Model (GSM) at T62L18 resolution (about 200km in horizontal and 18 vertical sigma levels) and the initial condition perturbations (2 pairs perturbed and 1 control members) were generated by breeding vector (BV) method (Toth and Kalnay 1993; Toth and Kalnay 1997; Toth et al. 1997; Toth et al. 2001; Zhu et al. 2002; Buizza et al. 2005; Zhu 2005). The GEFS ran once per day, out to 12 days in the early 90s. During the early 2000s, the 1st generation of GEFS reforecast (1979 - 2006) was produced off-line from using NCEP GFS/GEFS 1998 model version by NOAA PSL (Hamill et al. 2006) to demonstrate the improved ensemble reliability through bias correction and calibration. Over the years, the GEFS has been upgraded. In early 2010, the GEFS was upgraded with enhanced representation of model uncertainty using the Stochastic Total Tendency Perturbation (STTP) algorithm (Hou et al., 2008). The stochastic tendency perturbations were updated every 6 hours. Meanwhile, the 2nd generation of NOAA GEFS reforecasts were produced off-line for 29 years (1985 - 2013) by NOAA PSL (Hamill et al. 2013; NOAA/PSL reforecast website) using GEFS v10 configurations and CFS reanalysis. Through another major upgrade in December 2015, the GEFS initial perturbations were chosen from the operational hybrid Global Data Assimilation System (GDAS) 80-member Ensemble Kalman Filter (EnKF; Whitaker et al., 2008) 6-h forecasts along with tropical storm relocation and centralization of the initial perturbations (Zhou et al. 2016; 2017).More information on GEFS can be found at [a link ]. GEFS data can be found in the GEFS bucket: gs://gfs-ensemble-forecast-system Pub/Sub topics you can subscribe to for updates: projects/gcp-public-data-weather/topics/gfs-ensemble-forecast-system

This dataset is a cleaned-up extract from the following public BigQuery dataset: https://console.cloud.google.com/marketplace/details/noaa-public/ghcn-d

The dataset contains daily min/max temperatures from a selection of 1666 weather stations. The data spans exactly 50 years. Missing values have been interpolated and are marked as such.

This dataset is in TFRecord format.

About the original dataset: NOAA’s Global Historical Climatology Network (GHCN) is an integrated database of climate summaries from land surface stations across the globe that have been subjected to a common suite of quality assurance reviews. The data are obtained from more than 20 sources. The GHCN-Daily is an integrated database of daily climate summaries from land surface stations across the globe, and is comprised of daily climate records from over 100,000 stations in 180 countries and territories, and includes some data from every year since 1763.

The NOAA Monthly U.S. Climate Gridded Dataset (NClimGrid) consists of four climate variables derived from the GHCN-D dataset: maximum temperature, minimum temperature, average temperature and precipitation. Each file provides monthly values in a 5x5 lat/lon grid for the Continental United States. Data is available from 1895 to the present. On an annual basis, approximately one year of "final" nClimGrid will be submitted to replace the initially supplied "preliminary" data for the same time period. Users should be sure to ascertain which level of data is required for their research. This public dataset is hosted in Google Cloud Storage and available free to use. Use this quick start guide to quickly learn how to access public datasets on Google Cloud Storage.

GOES-17 (Geostationary Operational Environmental Satellite) is the second in the GOES-R series that promises significant upgrades in observing environmental phenomena. It provides images of weather patterns and severe storms as frequently as every 30 seconds, which supports more accurate and reliable weather forecasts and severe weather outlooks. The dataset includes a feed of the Advanced Baseline Imager (ABI) radiance data (Level 1b) and Cloud and Moisture Imager (CMI) products (Level 2). The NOAA Big Data Project (BDP) is an experimental collaboration between NOAA and infrastructure-as-a-service (IaaS) providers to explore methods of expanding the accessibility of NOAA’s data to facilitate innovation and collaboration. The goal is to help form new lines of business and facilitate economic growth while making NOAA's data more easily discoverable for the American public. This public dataset is hosted in Google Cloud Storage and available free to use. Click the "view dataset" button at the top to access the raw NetCDF files in Cloud Storage. Check out this quick start guide to learn how to access public datasets on Google Cloud Storage. This dataset includes a Pub/Sub topic you can subscribe to in order to be notified of updates. Subscribe to the topic 'projects/gcp-public-data---goes-17/topics/gcp-public-data-goes-17'. Use the Pub/Sub Quickstarts guide to learn more.

This dataset contains GIS layers that depict the known spatial distributions (i.e., ranges) and reported breeding areas of spotted seals (Phoca largha). It was produced as part of a U.S. Endangered Species Act status review, which included delineating the species in question and assessing its risk of extinction within the foreseeable future throughout all or a significant portion of its range. Its boundaries are based on previously published range maps and/or descriptions of the species' distribution in published or unpublished accounts. All boundaries should be considered approximate.Metadata accessed here:https://inport.nmfs.noaa.gov/inport/item/28183Data hosted as an online resource can be accessed here:https://console.cloud.google.com/storage/browser/nmfs_odp_afsc/MML/PEP/Spotted%20Seal%20Distribution

NOAA’s Global Historical Climatology Network (GHCN) is an integrated database of climate summaries from land surface stations across the globe that have been subjected to a common suite of quality assurance reviews. The data are obtained from more than 20 sources. Two GHCN datasets are available in BigQuery, the GHCN-D (daily) and the GHCN-M (monthly). The GHCN-Monthly is a temperature dataset that contains monthly mean temperatures and is used for operational climate monitoring activities. It is comprised of climate records from over 7,000 stations. For a complete description of data variables available in this dataset, see NOAA’s GHCN-M readme . This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery .

This dataset contains GIS layers that depict the known spatial distributions (i.e., ranges) and reported breeding areas of ribbon seals (Histriophoca fasciata). It was produced as part of a U.S. Endangered Species Act status review, which included delineating the species in question and assessing its risk of extinction within the foreseeable future throughout all or a significant portion of its range. Its boundaries are based on previously published range maps and/or descriptions of the species' distribution in published or unpublished accounts. All boundaries should be considered approximate.Metadata accessed here:https://inport.nmfs.noaa.gov/inport/item/28181Data hosted as an online resource can be accessed here:https://console.cloud.google.com/storage/browser/nmfs_odp_afsc/MML/PEP/Ribbon%20Seal%20Distribution

This dataset contains cloud-to-ground lightning strike information collected by Vaisala's National Lightning Detection Network and aggregated into 0.1 x 0.1 degree tiles by the experts at the National Centers for Environmental Information (NCEI) as part of their Severe Weather Data Inventory. This data provides historical cloud-to-ground data aggregated into tiles that around roughly 11 KMs for redistribution. This provides users with the number of lightning strikes each day, as well as the center point for each tile. The sample queries below will help you get started using BigQuery's GIS capabilities to analyze the data. For more on BigQuery GIS, see the documentation available here. The data begins in 1987 and runs through current day, with a delay of a few days for processing. For near real-time lightning information, see the Cloud Public Data's metadata listing of GOES-16 data for cloud-to-cloud and cloud-to-ground strikes over the eastern half of the western hemisphere. GOES-17 data covering the western half of the western hemisphere will be available soon. This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery .

This dataset contains GIS layers that depict the known spatial distributions (i.e., ranges) of the five subspecies of ringed seals (Phoca hispida). It was produced as part of a U.S. Endangered Species Act status review, which included delineating the species in question and assessing its risk of extinction within the foreseeable future throughout all or a significant portion of its range. Its boundaries are based on previously published range maps and/or descriptions of the species' distribution in published or unpublished accounts. All boundaries should be considered approximate.Metadata accessed here:https://inport.nmfs.noaa.gov/inport/item/28179Data hosted as an online resource can be accessed here:https://console.cloud.google.com/storage/browser/nmfs_odp_afsc/MML/PEP/Ringed%20Seal%20Distribution

This resource contains Jupyter Python notebooks which are intended to be used to learn about the U.S. National Water Model (NWM). These notebooks explore NWM forecasts in various ways. NWM Notebooks 1, 2, and 3, access NWM forecasts directly from the NOAA NOMADS file sharing system. Notebook 4 accesses NWM forecasts from Google Cloud Platform (GCP) storage in addition to NOMADS. A brief summary of what each notebook does is included below:

Notebook 1 (NWM1_Visualization) focuses on visualization. It includes functions for downloading and extracting time series forecasts for any of the 2.7 million stream reaches of the U.S. NWM. It also demonstrates ways to visualize forecasts using Python packages like matplotlib.

Notebook 2 (NWM2_Xarray) explores methods for slicing and dicing NWM NetCDF files using the python library, XArray.

Notebook 3 (NWM3_Subsetting) is focused on subsetting NWM forecasts and NetCDF files for specified reaches and exporting NWM forecast data to CSV files.

Notebook 4 (NWM4_Hydrotools) uses Hydrotools, a new suite of tools for evaluating NWM data, to retrieve NWM forecasts both from NOMADS and from Google Cloud Platform storage where older NWM forecasts are cached. This notebook also briefly covers visualizing, subsetting, and exporting forecasts retrieved with Hydrotools.

NOTE: Notebook 4 Requires a newer version of NumPy that is not available on the default CUAHSI JupyterHub instance. Please use the instance "HydroLearn - Intelligent Earth" and ensure to run !pip install hydrotools.nwm_client[gcp].

The notebooks are part of a NWM learning module on HydroLearn.org. When the associated learning module is complete, the link to it will be added here. It is recommended that these notebooks be opened through the CUAHSI JupyterHub App on Hydroshare. This can be done via the 'Open With' button at the top of this resource page.

The Earth Observations Group (EOG) at National Oceanic and Atmospheric Administration (NOAA)/National Geophysical Data Center (NGDC) is producing a version 1 suite of average radiance composite images using nighttime data from the Visible Infrared Imaging Radiometer Suite (VIIRS) Day/Night Band (DNB). Prior to averaging, the DNB data is filtered to exclude data impacted by stray light, lightning, lunar illumination, and cloud-cover. Cloud-cover is determined using the VIIRS Cloud Mask product (VCM). In addition, data near the edges of the swath are not included in the composites (aggregation zones 29-32). Temporal averaging is done on a monthly and annual basis. The version 1 series of monthly composites has not been filtered to screen out lights from aurora, fires, boats, and other temporal lights. However, the annual composites have layers with additional separation, removing temporal lights and background (non-light) values. The version 1 products span the globe from 75N latitude to 65S. The products are produced in 15 arc-second geographic grids and are made available in geotiff format as a set of 6 tiles. The tiles are cut at the equator and each span 120 degrees of latitude. Each tile is actually a set of images containing average radiance values and numbers of available observations. The dataset is the night light annual composite in year of 2015. The dataset is a KML file which requires the Google earth to visualize. For other monthly and annual basis night light geotiff datasets (up to Sep 2017), please download at https://www.ngdc.noaa.gov/eog/viirs/download_dnb_composites.html#NTL_2015 Citation: the Earth Observation Group, NOAA National Geophysical Data Center

This dataset contains GIS layers that depict the known spatial distributions (i.e., ranges) of the two subspecies of bearded seals (Erignathus barbatus). It was produced as part of a U.S. Endangered Species Act status review, which included delineating the species in question and assessing its risk of extinction within the foreseeable future throughout all or a significant portion of its range. Its boundaries are based on previously published range maps and/or descriptions of the species' distribution in published or unpublished accounts. All boundaries should be considered approximate. The approximate North American boundary between the two sub-species was changed to 130W (from 112W), based a re-analysis of the genetic data.Metadata accessed here:https://inport.nmfs.noaa.gov/inport/item/28177Data hosted as an online resource can be accessed here:https://console.cloud.google.com/storage/browser/nmfs_odp_afsc/MML/PEP/Bearded%20Seal%20Distribution

The High-Resolution Rapid Refresh (HRRR) is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh. For more information, see the HRRR Info Page from NOAA ESRL. In addition to the real-time data that is continuously updated, archived data is now available for HRRR forecasts. This data dates back as far as 2014, and is one of the most complete publicly-available archives of HRRR data. This dataset includes a Pub/Sub topic you can subscribe to in order to be notified of updates. Subscribe to the topic 'projects/gcp-public-data-weather/topics/gcp-public-data-hrrr'. Use the Pub/Sub Quickstarts guide to learn more. This public dataset is hosted in Google Cloud Storage and available free to use. Use this quick start guide to quickly learn how to access public datasets on Google Cloud Storage.

The Global Forecast System (GFS) is a weather forecast model produced by the National Centers for Environmental Prediction (NCEP). The GFS dataset consists of selected model outputs (described below) as gridded forecast variables. The 384-hour forecasts, with 1-hour (up to 120 hours) and 3-hour (after 120 hours) forecast intervals, are …

The Global Forecast System (GFS) is a weather forecast model produced by the National Centers for Environmental Prediction (NCEP). The GFS dataset consists of selected model outputs (described below) as gridded forecast variables. The 384-hour forecasts, with 3-hour forecast interval, are made at 6-hour temporal resolution (i.e. updated four times daily). Use the 'creation_time' and 'forecast_time' properties to select data of interest. The GFS is a coupled model, composed of an atmosphere model, an ocean model, a land/soil model, and a sea ice model which work together to provide an accurate picture of weather conditions. See history of recent modifications to the global forecast/analysis system , the model performance statistical web page , and the documentation homepage for more information.了解详情