This Version 7 NOAA Fundamental Climate Data Record (CDR) from Remote Sensing Systems (RSS) contains brightness temperatures that have been inter-calibrated and homogenized over the observation time period. The temperature data are from the Special Sensor Microwave Imager (SSM/I) and Special Sensor Microwave Imager Sounder (SSMIS) series of passive microwave radiometers carried onboard the Defense Meteorological Satellite Program (DMSP) satellites. These satellite sensors measure the natural microwave emission coming from the Earth’s surface in the spectral band from 19 to 85 GHz. This dataset encompasses data from a total of seven satellites including the SSM/I sensors on board DMSP satellites F08, F10, F11, F13, F14, and F15 as well as the SSMIS sensors on board DMSP satellite F17. The data record covers the time period from July 1987 through the present with a one month latency. The spatial and temporal resolutions of the CDR files correspond to the original resolution of the source SSMI(S) observations. There are roughly 15 orbits per day with a swath width of approximately 1400 km resulting in nearly global daily coverage. The spatial resolution of the data is a function of the sensor/channel and varies from approximately 50 km for the lowest frequency channels to approximately 15km for the high-frequency channels. The output parameters include the observed brightness temperatures for each of the seven SSM/I channels and 24 SSMIS channels at the original sensor channel resolution along with latitude and longitude information, time, quality flags, and view angle information. The file format is netCDF-4 with added metadata that follow the Climate and Forecast (CF) Conventions and Attribute Convention for Dataset Discovery (ACDD). There are three major changes in the Version 7 processing: (1) the water vapor continuum absorption model was re-derived, (2) the clear-sky bias in cloud water was removed and the data format for cloud water was changed, and (3) the beamfilling correction in the rain algorithm was modified. Relative to Version 6, Version 7 has: (1) increased vapor values in the range of 50-60 mm by 1%, (2) increased vapor values above 60 mm by 2-3%, (3) cloud data changed to the range of cloud water values: -0.05 to 2.45 mm (cloud data format has changed), and (4) increased the global mean rain rates by about 16% (mostly due to changes in the extratropical values).

Note: This dataset version has been superseded by a newer version. It is highly recommended that users access the current version. Users should only use this version for special cases, such as reproducing studies that used this version. The Special Sensor Microwave Imagers (SSM/I) are a series of six satellite radiometers that have been in operation since 1987 under the Defense Meteorological Satellite Program (DMSP). The six SSM/Is (aboard F08, F10, F11, F13, F14, and F15) have a seven channel linearly polarized passive microwave radiometer that operate at frequencies of 19.36 (vertical and horizontal polarized), 22.235 (vertical polarized), 37.0 (vertical and horizontal polarized), and 85.5 GHz (vertical and horizontal polarized). The Remote Sensing Systems (RSS) Version-6 SSM/I Fundamental Climate Data Record (FCDR) dataset has incorporated all past geolocation corrections, sensor calibration (including cross-scan biases), and quality control procedures in a consistent way for the entire 24-year SSM/I brightness temperature period of record. Version-5 was relatively short lived due to subtle calibration problems that caused small spurious trends in the climate retrievals (the SSM/I record had become long enough at this point to detect such errors). The problem was due to subtle correlations in the derivation of the target factors for the F10 and F14 SSM/I. Like the Microwave Sounding Unit (MSU), some of the SSM/I exhibit errors that are correlated with the hot-load target temperatures, and we removed these errors using the target multiplier approach. Application of the solutions described herein provided the current V6 SSM/I TA and TB dataset. RSS Version-6 SSM/I FCDR data are stored as netCDF-4 files that have been internally compressed at the maximum GZIP utility level. A typical file will have a size of 6.4 megabytes.

What is NOAA?

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2Fdd4a2c5f6708f178dedeaecb9eedeba4%2F_25c02555-8eb6-4839-8176-e747400f6cb9-small.jpeg?generation=1756854638071696&alt=media" alt="">

The National Oceanic and Atmospheric Administration (abbreviated as NOAA /ˈnoʊ.ə/ NOH-ə) is a US scientific and regulatory agency charged with forecasting weather, monitoring oceanic and atmospheric conditions, charting the seas, conducting deep-sea exploration, and managing fishing and protection of marine mammals and endangered species in the US exclusive economic zone. The agency is part of the United States Department of Commerce and is headquartered in Silver Spring, Maryland.

From Wikipedia

What is AIS data?

The automatic identification system, or AIS, transmits a ship’s position so that other ships are aware of its position. The International Maritime Organization and other management bodies require large ships, including many commercial fishing vessels, to broadcast their position with AIS in order to avoid collisions. Each year, more than 400,000 AIS devices broadcast vessel location, identity, course and speed information. Ground stations and satellites pick up this information, making vessels trackable even in the most remote areas of the ocean.

https://globalfishingwatch.org/faqs/what-is-ais/

NOAA AIS Data

Vessel traffic data, or Automatic Identification System (AIS) data, are collected by the U.S. Coast Guard through an onboard navigation safety device that transmits the location and characteristics of large vessels for tracking in real time. The MarineCadastre.gov project team has worked with the Coast Guard and NOAA’s Office of Coast Survey to repurpose and make available some of the most important data for use in ocean planning applications.

From https://coast.noaa.gov/digitalcoast/training/ais.html

Data Dictionary

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F1af61e6fe2afbb698f6af58085f5637a%2FAISDataDictionary.png?generation=1719076791476332&alt=media" alt="">

Other Data Dictionaries

• MMSI Country code prefixes
• Navigational Status Codes
• Ship Types
• Vessel Types

Source

https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2025/index.html

Notes

• Data is still incomplete for 2025 as the year is still not done
• Original data consists of daily data and was 100GB+. What I did was I separated the ship from the AIS data, plus broke it down per month of a year for a smaller footprint. I also converted it to parquet file format.
• There are some erroneous lines that are being encountered when importing. I removed them but this is like very, very, very, small that it's insignificant

Citations

I did not create the dataset

I just made the data more accessible and easier to utilize from an end user's perspective. All credits to NOAA and the AIS methodology.

For citation of NOAA, go here

More info here

Image

Image Generated with Bing

See the whole series

• 2014 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2015 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2016 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2017 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2018 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2019 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2020 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2021 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2022 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2023 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2024 🇺🇸 US Coast Guard + NOAA AIS Dataset

This dataset contains the bias-corrected CPC MORPHing technique (CMORPH) global precipitation analyses, version 1, and is obtained from the NOAA Climate Data Record [https://www.ncei.noaa.gov/products/climate-data-records/precipitation-cmorph].

The following description is from the NOAA Climate Data Record CMORPH dataset page:

This data set is for the bias-corrected, reprocessed CPC Morphing technique (CMORPH) high-resolution global satellite precipitation estimates. The CMORPH satellite precipitation estimates are created in two steps. First, the purely satellite-based global fields of precipitation are constructed through integrating Level 2 retrievals of instantaneous precipitation rates from all available passive microwave measurements aboard low earth orbiting platforms. Bias in these integrated satellite precipitation estimates is then removed through comparison against CPC daily gauge analysis over land and adjustment against the Global Precipitation Climatology Project (GPCP) merged analysis of pentad precipitation over ocean. The bias corrected CMORPH satellite precipitation estimates are created on an 8 km by 8 km grid over the global domain from 60 degrees S to 60 degrees N and in a 30-minute interval from January 1, 1998. Due to the delay of some input data sets, this formal version (Version 1) bias corrected CMORPH is produced manually once a month at a latency of 3-4 months.

For the CDR production, the bias corrected CMORPH generated at its native resolution of 8 km by 8 km / 30-minute is upscaled to form THREE sets of data files of different time/space resolution for improved user experience:

a) the full-resolution CMORPH data Output variable: precipitation rate in mm/hour spatial resolution: 8 km by 8km (at equator) spatial coverage: global (60S-60N) temporal resolution: 30min data period: January 1, 1998 to the present

b) Hourly CMORPH Output variable: precipitation rate in mm/hour spatial resolution: 0.25 degrees latitude/longitude spatial coverage: global (60S-60N) temporal resolution: hourly data period: January 1, 1998 to the present

c) Daily CMORPH Output variable: daily precipitation in mm/day spatial resolution: 0.25 degrees latitude/longitude spatial coverage: global (60S-60N) temporal resolution: hourly data period: January 1, 1998 to the present

(b) and (c) are derived from and quantitatively consistent with the CMORPH at its original resolution (a).

This dataset contains Level 1c inter-calibrated brightness temperatures from the Microwave Sounding Unit (MSU) sensors onboard nine polar orbiting satellites (TIROS-N, NOAA-6, -7, -8, -9, -10, -11, -12, and -14) spanning from 1978 to 2006. The dataset was produced by the NOAA Center for Satellite Applications and Research (STAR), and is a Fundamental Climate Data Record (FCDR) of microwave brightness temperatures in the NOAA CDR Program. MSU is a four-channel microwave radiometer measuring at 50.3, 53.74, 54.96, and 57.95 GHz, and has ground spatial resolution of about 250 km in diameter at nadir. The native MSU Level 1b data were inter-calibrated using the Integrated Microwave Inter-Calibration Approach (IMICA) method to obtain a long-term data product to be used in climate analyses. For comparison, data files also include the operational data used in NWP forecasting along with the IMICA calibrated radiances, which minimize or remove the biases found in the operational calibration. In addition, limb adjusted radiances for both the IMICA and operational calibrations are included for certain type of climate applications, such as atmospheric layer temperature development using the radiance datasets. The orbital swath data files include MSU channels 2 through 4 for the IMICA calibration, and channels 1 through 4 for the operational calibration. The inter-calibrated MSU data are not expected to change for the dataset time period.

The Infrared Atmospheric Sounding Interferometer is a Cross-nadir infrared sounder flown on the European MetOp satellite series. IASI measures atmospheric emission spectra to derive temperature and humidity profiles with high vertical resolution and accuracy. IASI is a Michelson interferometer with spectral coverage between 3.6 and 15.5 micrometers. At nadir, the instrument samples data at intervals of 25 km along track and cross track with each sample having a maximum diameter of about 12 km. The IASI Level-2 Cloud-Cleared Radiances (CCR) product from MetOp-B/C is a NOAA-unique product generated from the European Organization for Meteorological Satellites (EUMETSAT) IASI level 1C data. The data are produced by the NOAA Environmental Satellite, Data, and Information Service (NESDIS) and are distributed by the Comprehensive Large Array-Data Stewardship System (CLASS) as 3 minute files in the netCDF-4 file format with attributes included.

Dataset Description:

This dataset compiles and standardizes climate-related data from multiple authoritative sources to analyze factors influencing Global Mean Sea Level (GMSL). It includes key environmental indicators such as CO₂ concentrations, greenhouse gas emissions, temperature anomalies, sea surface temperatures, El Niño indices, and polar sea ice extent. The dataset spans 1880-2024, making it suitable for long-term trend analysis and time series forecasting models like LSTM, Bi-LSTM, ARIMA, and SARIMA.

Data Collection & Processing:

The data was sourced from the following organizations and repositories:
- NOAA (ncei.noaa.gov)
- Our World in Data (ourworldindata.org)
- Copernicus Climate Data Store (cds.climate.copernicus.eu)
- National Snow & Ice Data Center (NSIDC) (nsidc.org)
- NASA Earth Science (earth.gsfc.nasa.gov)
- ENSO Monitoring - NOAA (ncei.noaa.gov)
- Permanent Service for Mean Sea Level (PSMSL) (psmsl.org)
- NOAA Climate Prediction Center (CPC) (origin.cpc.ncep.noaa.gov)
- NOAA Physical Sciences Laboratory (PSL) (psl.noaa.gov)

Features:

The dataset includes the following variables:
- Date (1880-2024)
- Global Mean Sea Level (GMSL)
- CO₂ concentration & emissions
- Long-run NO₂ & CH₄ concentration/emissions
- Global average temperature anomaly
- Sea surface temperature anomaly & trend
- El Niño indices (Nino 1.2, Nino 3, Nino 3.4, Nino 4) and trends
- North & South Pole sea ice extent (avg, min, max, trends)

Data Processing Steps:

1. Cleaning & Imputation: Removed corrupted data and filled missing values using mean interpolation.
2. Time Standardization: Unified different time series (e.g., 1978-2020, 1996-2008) into a single continuous series from 1880-2024 using linear and polynomial extrapolation.
3. Temporal Aggregation: Converted daily and seasonal values into annual means for consistency.
4. Multi-Satellite Data Handling: Averaged readings from different satellites where multiple observations were available.
5. Seasonal Variability Adjustments: Created trends and anomalies for seasonally varying data.
6. Exploratory Data Analysis (EDA) & Feature Selection: Conducted feature engineering and selected key predictors for forecasting.
7. Final Dataset: Ready for application in time series forecasting models like LSTM, Bi-LSTM, ARIMA, SARIMA, using the Date column as the primary time index.

Acknowledgments:

This dataset is compiled using publicly available data from NOAA, Our World in Data, Copernicus Climate Data Store, NSIDC, NASA, PSMSL, and NOAA Climate Monitoring Centers. Please cite the respective sources when using this dataset in research or analysis.

Release of the hourly database containing 55 cruises spanning over 31 years, including the historical data from 12 cruises done in the 1990's (Fairall et al., 2003) and 9 cruises of the PACS/EPIC dataset. Data collected from these cruises are critical for supporting the study of physical oceanography, air-sea interaction, tropical meteorology, as well as global weather and climate variability and predictability. This includes improvement to our fundamental understanding of these processes in the ocean and their influence around the globe including the Continental United States. The data will also support improvement and validation of prediction models including parameterizations. Sensible heat flux was computed from vertical velocity - sonic temperature covariance. The humidity contribution to sonic temperature was removed using the bulk latent heat flux. acknowledgement=NOAA Global Ocean Monitoring and Observing (GOMO) program cdm_data_type=Trajectory cdm_trajectory_variables=cruise_name comment=Corrections and Data Quality Notes not contained in global or variable attributes: Unavailable data, bad data, and data within restricted Exclusive Economic Zones were assigned _FillValue = -9999. Please use the variables named flag_bad_ship and flag_bad_bulk to further mask out questionable or non-ideal data points depending on the application for state variables and bulk fluxes respectively. comment2=Sensible heat flux was computed from vertical velocity - sonic temperature covariance. The humidity contribution to sonic temperature was removed using the bulk latent heat flux. comment3=A correction to account for biases in gas concentration measurements has been applied on the covariance and ID latent heat fluxes. See Fratini et al. 2014 for more details. comment4=Data from the 2004 New England Air Quality Study (NEAQS-04) are included in this dataset but it has to be noted that during that project we found significant suppression of the transfer coefficients for momentum, sensible heat, and latent heat; mainly because our measurements at 18-m height did not realize the full surface flux in these shallower boundary layer conditions. (Fairall et al., 2006). Conventions=CF-1.6, ACCD-1.3, COARDS, ACDD-1.3 coverage_content_type=physicalMeasurement, qualityInformation, modelResult, coordinate date_metadata_modified=2023-04-18T13:10:40Z Easternmost_Easting=179.73351 featureType=Trajectory geospatial_lat_bounds=POLYGON [-179.833, 179.734, -53.754, 69.934] geospatial_lat_max=69.933717 geospatial_lat_min=-53.753807 geospatial_lat_units=degrees_north geospatial_lon_max=179.73351 geospatial_lon_min=-179.83283 geospatial_lon_units=degrees_east geospatial_vertical_max=0.014330280134111055 geospatial_vertical_min=1.7080496969024725E-4 geospatial_vertical_positive=down geospatial_vertical_units=m history=v0: original data, v1: first release id=doi = not yet assigned infoUrl=https://psl.noaa.gov/boundary-layer/ institution=(1) NOAA Physical Sciences Lab (PSL); (2) CIRES Cooperative Institute for Research in Environmental Sciences at the University of Colorado Boulder in partnership with NOAA PSL instrument_vocabulary=GCMD Version 12.3 keywords_library=GCMD Version 12.3 keywords_vocabulary=GCMD Science Keywords licence=Please acknowledge data according to global attribute info: acknowledgement, creator_name, creator_institution. These data may be redistributed and used without restriction. naming_authority=gov.noaa.ncei Northernmost_Northing=69.933717 platform=refer to platform_name variable that contains names of the different platforms from which the datasets were collected platform_vocabulary=GCMD Version 12.3 processing_level=processed and quality controlled program=Funding agencies: NOAA Global Ocean Monitoring and Observing (GOMO) program project=refer to cruise_name variable for project names of various datasets references=Fairall et al. 1996a JGR https://doi.org/10.1029/95JC03190 ...Fairall et al. 1996b JGR https://doi.org/10.1029/95JC03205 .... Fairall et al. 2003 JClim https://doi.org/10.1175/1520-0442(2003)016%3C0571:BPOASF%3E2.0.CO;2 ... Edson et al. 2013 JPO with corrigendum: the value should be m = 0.0017, and not m = 0.017 as originally appeared https://doi.org/10.1175/JPO-D-12-0173.1 ... Fratini et al. 2014https://doi.org/10.5194/bg-11-1037-2014 ... Fairall et al. 2006 https://doi.org/10.1029/2006JD007597 sea_name=Northwest, Equatorial and SouthEast Pacific Ocean; North Atlantic Ocean; Davis Strait; Labrador Sea; South Atlantic Ocean; Bay of Bengal; Indian Ocean; Tasman Sea source=observations from NOAA PSL sensors (no subscript, most accurate) and the ship permanent sensors (_ship subscript, less accurate), derivations from those observations using eddy covariance and inertial dissipation methods of estimating fluxes, model results from COARE 3.6 bulk air-sea flux algorithm. Wave parameters were not used as input to COARE since they were either unavailable or not consistently available on all projects. Also True water-relative wind speed was used as input to COARE when available. Otherwise when not available the true wind speed was used instead. sourceUrl=(local files) Southernmost_Northing=-53.753807 standard_name_vocabulary=CF Standard Name Table v70 subsetVariables=platform_call_sign, platform_name, cruise_name time_coverage_duration=31 years time_coverage_end=2021-08-31T23:00:00Z time_coverage_resolution=PT60.M time_coverage_start=1991-11-22T11:41:00Z Westernmost_Easting=-179.83283

This dataset contains Level 1c inter-calibrated brightness temperatures from the Advanced Microwave Sounding Unit-A (AMSU-A) sensors onboard six polar orbiting satellites (NOAA-15, -16, -17, -18, EUMETSAT MetOp-A, and NASA Aqua) spanning from 1998 to the present. The dataset was produced by the NOAA Center for Satellite Applications and Research (STAR), and is a Fundamental Climate Data Record (FCDR) of microwave brightness temperature in the NOAA CDR Program. AMSU-A is a 15-channel microwave radiometer with a ground spatial resolution of about 50 km in diameter at nadir. The native AMSU-A Level 1b data were inter-calibrated using the Integrated Microwave Inter-Calibration Approach (IMICA) method to obtain a long-term data product to be used in climate analyses. For comparison, data files also include the operational data used in NWP forecasting along with the IMICA calibrated radiances, which minimize or remove the biases found in the operational calibration. In addition, limb adjusted radiances for both the IMICA and operational calibrations are included for certain type of climate applications, such as atmospheric layer temperature development using the radiance datasets. The orbital swath data files include AMSU-A channels 4 through 14 (between 52.8 and 57.6 GHz) for both the IMICA calibration and the operational calibration. The inter-calibrated AMSU-A data may be updated as the sounding units are further calibrated over time.

In early 2017, a gridded version of the dataset was made available to the public. There are three types of gridding approaches: brightness temperatures of near-nadir Field of View (FOV) only, brightness temperature of FOV with minimum viewing zenith angle, and average brightness temperatures of FOVs from multiple scan positions.The gridded data are the produced using the same methodologies as the swath data, but are calculated to a 1x1 degree grid resolution with global coverage.

What is NOAA?

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2Fb92d3310793ab9b7a3f3e86bc91d05ab%2F_97f25c8f-dbd6-487d-8710-b3607aef8d50_COPY.jpeg?generation=1719046642408648&alt=media" alt="">

The National Oceanic and Atmospheric Administration (abbreviated as NOAA /ˈnoʊ.ə/ NOH-ə) is a US scientific and regulatory agency charged with forecasting weather, monitoring oceanic and atmospheric conditions, charting the seas, conducting deep-sea exploration, and managing fishing and protection of marine mammals and endangered species in the US exclusive economic zone. The agency is part of the United States Department of Commerce and is headquartered in Silver Spring, Maryland.

From Wikipedia

What is AIS data?

The automatic identification system, or AIS, transmits a ship’s position so that other ships are aware of its position. The International Maritime Organization and other management bodies require large ships, including many commercial fishing vessels, to broadcast their position with AIS in order to avoid collisions. Each year, more than 400,000 AIS devices broadcast vessel location, identity, course and speed information. Ground stations and satellites pick up this information, making vessels trackable even in the most remote areas of the ocean.

https://globalfishingwatch.org/faqs/what-is-ais/

NOAA AIS Data

Vessel traffic data, or Automatic Identification System (AIS) data, are collected by the U.S. Coast Guard through an onboard navigation safety device that transmits the location and characteristics of large vessels for tracking in real time. The MarineCadastre.gov project team has worked with the Coast Guard and NOAA’s Office of Coast Survey to repurpose and make available some of the most important data for use in ocean planning applications.

From https://coast.noaa.gov/digitalcoast/training/ais.html

Data Dictionary

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F4b109911f464f6a6cdf33016721c9da4%2F2017DataDictionary.png?generation=1719036980797581&alt=media" alt="">

Other Data Dictionaries

• MMSI Country code prefixes
• Navigational Status Codes
• Ship Types
• Vessel Types

Source

https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2016/index.html

Notes

• Original data consists of daily data and was 100GB+. What I did was I separated the ship from the AIS data, plus broke it down per month of a year for a smaller footprint. I also converted it to parquet file format.
• There are some erroneous lines that are being encountered when importing. I removed them but this is like very, very, very, small that it's insignificant

Citations

I did not create the dataset

I just made the data more accessible and easier to utilize from an end user's perspective. All credits to NOAA and the AIS methodology.

For citation of NOAA, go here

More info here

Image

Image Generated with Bing

See the whole series

• 2014 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2015 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2016 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2017 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2018 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2019 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2020 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2021 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2022 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2023 🇺🇸 US Coast Guard + NOAA AIS Dataset

What is NOAA?

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F97772da9539704a1b341a787913f235b%2F_cd11bdb6-6a55-4e4e-9ff9-359503e2a8a5-small.jpeg?generation=1738125724097651&alt=media" alt="">

The National Oceanic and Atmospheric Administration (abbreviated as NOAA /ˈnoʊ.ə/ NOH-ə) is a US scientific and regulatory agency charged with forecasting weather, monitoring oceanic and atmospheric conditions, charting the seas, conducting deep-sea exploration, and managing fishing and protection of marine mammals and endangered species in the US exclusive economic zone. The agency is part of the United States Department of Commerce and is headquartered in Silver Spring, Maryland.

From Wikipedia

What is AIS data?

The automatic identification system, or AIS, transmits a ship’s position so that other ships are aware of its position. The International Maritime Organization and other management bodies require large ships, including many commercial fishing vessels, to broadcast their position with AIS in order to avoid collisions. Each year, more than 400,000 AIS devices broadcast vessel location, identity, course and speed information. Ground stations and satellites pick up this information, making vessels trackable even in the most remote areas of the ocean.

https://globalfishingwatch.org/faqs/what-is-ais/

NOAA AIS Data

Vessel traffic data, or Automatic Identification System (AIS) data, are collected by the U.S. Coast Guard through an onboard navigation safety device that transmits the location and characteristics of large vessels for tracking in real time. The MarineCadastre.gov project team has worked with the Coast Guard and NOAA’s Office of Coast Survey to repurpose and make available some of the most important data for use in ocean planning applications.

From https://coast.noaa.gov/digitalcoast/training/ais.html

Data Dictionary

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1842206%2F1af61e6fe2afbb698f6af58085f5637a%2FAISDataDictionary.png?generation=1719076791476332&alt=media" alt="">

Other Data Dictionaries

• MMSI Country code prefixes
• Navigational Status Codes
• Ship Types
• Vessel Types

Source

https://coast.noaa.gov/htdata/CMSP/AISDataHandler/2024/index.html

Notes

• Original data consists of daily data and was 100GB+. What I did was I separated the ship from the AIS data, plus broke it down per month of a year for a smaller footprint. I also converted it to parquet file format.
• There are some erroneous lines that are being encountered when importing. I removed them but this is like very, very, very, small that it's insignificant

Citations

I did not create the dataset

I just made the data more accessible and easier to utilize from an end user's perspective. All credits to NOAA and the AIS methodology.

For citation of NOAA, go here

More info here

Image

Image Generated with Bing

See the whole series

• 2014 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2015 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2016 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2017 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2018 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2019 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2020 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2021 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2022 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2023 🇺🇸 US Coast Guard + NOAA AIS Dataset
• 2024 🇺🇸 US Coast Guard + NOAA AIS Dataset

This dataset contains dropsondes dropped from the NOAA N49 aircraft as part of the Hurricane Rainband and Intensity Change Experiment (RAINEX). It includes 516 high resolution (.5 sec) soundings dropped from July 6 - Oct 24, 2005 during hurricanes Dennis, Emily, Katrina, Ophelia, Rita, and Wilma.

The climate data are related to Albany and they cover a period that goes from 01/01/2015 to 05/31/2022. They include wind, temperature, pressure, humidity and precipitation data. Four datasets are included:

• daily_data.csv contains all the daily data.
• hourly_data.csv with the hourly data.
• monthly_data.csv includes data for each month.
• three_hour_data.csv where data were collected every three hours.

** **

For more information, check out here: https://www.ncei.noaa.gov/pub/data/cdo/documentation/LCD_documentation.pdf.

The following values can be encountered: s = suspect value (appears together with value). T = trace precipitation amount or snow depth (an amount too small to measure, usually < 0.005 inches water equivalent) (appears instead of numeric value). M = missing value (appears instead of value). VRB = variable wind direction. Remember to upvote if you found the dataset useful :).

Inspiration

The dataset can be used to perform supervised learning to predict one of the numerical features in the dataset, given a set of selected input features. You can perform an exploratory data analysis of the data, working with Pandas or Numpy(if you use Python).

Interesting visualizations can be performed using, for instance, Python libraries like Matplotlib. A time series analysis and forecasting can be performed too. Moreover, this dataset is very good to practice queries using SQL or Pandas.

Collection methodology

The data were fetched from NCOI website. The data were split in 4 columns according to the REPORT_TYPE. Rows containing null values were dropped and empty or partially empty columns were not considered.

Acknowledgment

DOC/NOAA/NESDIS/NCEI > National Centers for Environmental Information, NESDIS, NOAA, U.S. Department of Commerce DOC/NOAA/NWS > National Weather Service, NOAA, U.S. Department of Commerce DOD/USAF > U.S. Air Force, U.S. Department of Defense DOT/FAA > Federal Aviation Agency, U.S. Department of Transportation

This dataset contains dropsondes dropped from the NOAA N43 aircraft as part of the Hurricane Rainband and Intensity Change Experiment (RAINEX). It includes 273 high resolution (.5 sec) soundings dropped from August 25 - Sept 23, 2005.

This dataset contains dropsondes dropped from the NOAA N42 aircraft while sampling Hurricane Ophelia and Hurricane Rita as part of the Hurricane Rainband and Intensity Change Experiment (RAINEX). It includes 124 high resolution (.5 sec) soundings dropped on Sept 7, 9, 11, 12, 16, 17, 22, and 23, 2005.

The National Climatic Data Center Tape Deck Documentation library is a collection of over 400 manuals describing NCDC's digital holdings (both historic and current). While many libraries are still available from NCDC, some datasets have been removed due to obsolesence--their respective manuals remain to serve as a permanent record of the dataset.

The BASE Temperature Data Record (TDR) dataset from Colorado State University (CSU) is a collection of the raw unprocessed antenna temperature data that has been written into single orbit granules and reformatted into netCDF-4. The temperature data are from the Special Sensor Microwave Imager (SSM/I) and Special Sensor Microwave Imager Sounder (SSMIS) series of passive microwave radiometers carried onboard the Defense Meteorological Satellite Program (DMSP) satellites. This dataset encompasses data from a total of nine satellites including the SSM/I sensors on board DMSP satellites F08, F10, F11, F13, F14, and F15 as well as the SSMIS sensors on board DMSP satellites F16, F17, and F18. The data record covers the time period from July 1987 through the present with a 7 to 10 day latency. The spatial and temporal resolutions of the BASE files correspond to the original resolution of the raw source TDR observations. There are roughly 15 orbits per day with a swath width of approximately 1400 km resulting in nearly global daily coverage. The spatial resolution of the data is a function of the sensor/channel and varies from approximately ~50 km for the lowest frequency channels to ~15km for the high-frequency channels. These files contain all of the information from the original source TDR files with the following changes/additions. The BASE files have been reorganized into single orbit granules with duplicate scans removed, and spacecraft position and velocity based on the TLE (two line element) data have been added for calculating geolocation. With the exception of duplicate scans, none of data from the original TDR files was changed or removed. This BASE TDR dataset is used by CSU as input for the subsequent processing of the final intercalibrated Fundamental Climate Data Record (FCDR). The file format is netCDF-4 with added metadata that follow the Climate and Forecast (CF) Conventions and Attribute Convention for Dataset Discovery (ACDD).

Greenhouse Gases Data Visualization with Matplotlib

Steps:

- Create One Datasets combining txt and csv datasets
- Feature engineering with creatinng new variables from exists
- Line, Bar, Histogram, Errorbar, Box, Scatter, Stem plots examples

• I created this Datasetcombining NOAA - Global Monitoring Laboratory's studies on greenhouse gases

• Dataset has nearly two decade data

• Dataset is about monthly measuring the atmospheric distribution and trends of the main long-term drivers of climate change, carbon dioxide (CO2), methane (CH4), and nitrous oxide (N2O), as well as Sulfur hexafluoride (SF6).

Variables

relative_temp : relative temperature values according to reference temperature

relative_co2 : relative co2 value according to next measurement

relative_ch4 : relative ch4 value according to next measurement

relative_n2o : relative n2o value according to next measurement

relative_sf6 : relative sf6 value according to next measurement

CO2: Data are reported as a dry air mole fraction defined as the number of molecules of carbon dioxide divided by the number of all molecules in air, including CO2 itself, after water vapor has been removed. The mole fraction is expressed as parts per million (ppm). Example: 0.000400 is expressed as 400 ppm.

CH4: Methane is reported as a “dry air mole fraction”, defined as the number of molecules of methane divided by the total number of molecules in the sample, after water vapor has been removed. The mole fraction is expressed as nmol mol-1, abbreviated “ppb” (for parts per billion; 1 ppb indicates that one out of every billion molecules in an air sample is CH4).

N2O: Nitrous oxide is reported as a “dry air mole fraction”, defined as the number of molecules of nitrous oxide divided by the total number of molecules in the sample, after water vapor has been removed. The mole fraction is expressed as nmol mol-1, abbreviated “ppb” (for parts per billion; 1 ppb indicates that one out of every billion molecules in an air sample is N2O).

SF6: Sulfur hexafluoride is reported as a “dry air mole fraction”, defined as the number of molecules of sulfur hexafluoride divided by the total number of molecules in the sample, after water vapor has been removed. The mole fraction is expressed as pmol mol-1, abbreviated “ppt” (for parts per trillion; 1 ppt indicates that one out of every trillion molecules in an air sample is SF6).

Download Dataset and Work files

• to get dataset :

https://github.com/hakansaritas/Create-greenhouse-gases-dataset-and-visualize/blob/main/greenhouse_gases.csv

• to figure out how to create dataset

https://github.com/hakansaritas/Create-greenhouse-gases-dataset-and-visualize/blob/main/Create_Dataset.ipynb

• to reach this jupyternotebook work file

https://github.com/hakansaritas/Create-greenhouse-gases-dataset-and-visualize/blob/main/matplotlib_analysis.ipynb

For More Detail about data: https://gml.noaa.gov/ccgg/

https://www.ncei.noaa.gov/access/monitoring/global-temperature-anomalies/

The NOAA Office for Coastal Management’s composite shoreline is a high-resolution vector shoreline based on a multi-temporal collection of NOAA shoreline manuscripts (T-sheets), which are special-use planimetric or topographic maps that precisely define the shoreline and alongshore natural and man-made features. Note that shorelines may have eroded, acreted, or been anthropogenically altered since the historic T-sheets were produced.These data are derived from shoreline data that were produced by the NOAA National Ocean Service including its predecessor agencies. Where T-sheets are unavailable, NOAA’s extracted vector shoreline (EVS) was used to compile seamless shoreline coverage. For this layer, NOAA's national dataset was clipped to the Florida state boundary and control points, jetties, and piers were removed.

AusENDVI (Australian Emprical NDVI) is a monthly, 5-km gridded estimate of NDVI across Australia from 1982-2022. It is built by calibrating and harmonising NOAA's Climate Data Record AVHRR NDVI data to MODIS MCD43A4 NDVI using a gradient boosting ensemble decision tree method. Additionally, the datasets are gapfilled using a synthetic NDVI dataset. The methods are extensively described in an Earth System Science Data pre-publication.

AusENDVI consists of several datasets, each dataset has a description in the attributes of the NetCDF file that describes its provenance. The naming convention is "AusENDVI_

1. AusENDVI-clim_1982_2013. Calibrated and harmonised NOAA's Climate Data Record AVHRR NDVI data from Jan. 1982 to Dec. 2013. This version of the dataset used climate data in the calibration and harmonisation process. The dataset has not been gap filled, and extra data has been filtered/removed beyond the typical QA filtering using methods described in the publication.
2. AusENDVI-noclim_1982_2013. Calibrated and harmonised NOAA's Climate Data Record AVHRR NDVI data from Jan. 1982 to Dec. 2013. This version of the dataset did not use climate data in the calibration and harmonisation process. The dataset has not been gap filled, and extra data has been filtered/removed beyond the typical QA filtering using methods described in the publication.
3. AusENDVI-synthetic_1982_2022. This dataset consists of synthetic NDVI data that was built by training a model on the joined _AusENDVI-clim_ and _MODIS-MCD43A4 NDVI_ timeseries using climate, woody-cover-fraction, and atmospheric CO2 as predictors.
4. AusENDVI-clim_gapfilled_MCD43A4_1982_2022. This dataset consists of calibrated and harmonised NOAA's Climate Data Record AVHRR NDVI data from Jan. 1982 to Feb. 2000, joined with MODIS-MCD43A4 NDVI data from Mar. 2000 to Dec. 2022. This version of the dataset _used climate data_ in the calibration and harmonisation process. The dataset has been gap filled using _AusENDVI-synthetic_
5. AusENDVI-noclim_MCD43A4_1982_2022. This dataset consists of calibrated and harmonised NOAA's Climate Data Record AVHRR NDVI data from Jan. 1982 to Feb. 2000, and MODIS-MCD43A4 NDVI data from Mar. 2000 to Dec. 2022. This version of the dataset _did not use climate data_ in the calibration and harmonisation process. The dataset has not been gap filled.

All datasets are in "EPSG:4326" projection, and have a spatial resolution of 0.05 degrees. Geographic coordinate information is contained in the "spatial_ref" variable.

A Jupyter Notebook is also provided that shows how to load, plot, QC mask, reproject, and gap-fill AusENDVI datasets. The notebook is effectively a 'readme' file.

• The notebook is also available to view/download here

An open-source github repository details the methods used to create these datasets

• https://github.com/cbur24/AusENDVI