This dataset contains the Coded Surface Bulletin (CSB) dataset reformatted as netCDF-4 files. The CSB dataset is a collection of ASCII files containing the locations of weather fronts, troughs, high pressure centers, and low pressure centers as determined by National Weather Service meteorologists at the Weather Prediction Center (WPC) during the surface analysis they do every three hours. Each bulletin is broadcast on the NOAAPort service, and has been available since 2003.

Each netCDF file contains one year of CSB fronts data represented as spatial map data grids. The times and geospatial locations for the data grid cells are also included. The front data is stored in a netCDF variable with dimensions (time, front type, y, x), where x and y are geospatial dimensions. There is a 2D geospatial data grid for each time step for each of the 4 front types—cold, warm, stationary, and occluded. The front polylines from the CSB dataset are rasterized into the appropriate data grids. Each file conforms to the Climate and Forecast Metadata Conventions.

There are two large groupings of the CSB netCDF files. One group uses a data grid based on the North American Regional Reanalysis (NARR) grid, which is a Lambert Conformal Conic projection coordinate reference system (CRS) centered over North America. The NARR grid is quite close the the spatial range of data displayed on the WPC workstations used to perform surface analysis and identify front locations. The native NARR grid has grid cells which are 32 km on each side. Our grid covers the same extents with cells that are 96 km on each side.

The other group uses a 1° latitude/longitude data grid centered over North America with extents 171W – 31W / 10N – 77 N. The files in this group are identified by the name MERRA2, because they were used with data from the NASA MERRA-2 dataset, which uses a latitude/longitude data grid.

There are a number of files within each group. The files all follow the naming convention codsus_[masked]_

The codsus_

The

Within each grid group, there are five subsets of files:

codsus_masked_

codsus_masked_

codsus_

codsus_

codsus_

The primary source for this dataset is an internal archive maintained by personnel at the WPC and provided to the author. It is also provided at DOI 10.5281/zenodo.2642801. Some bulletins missing from the WPC archive were filled in with data acquired from the Iowa Environmental Mesonet.

Data Summary: US states grid mask file and NOAA climate regions grid mask file, both compatible with the 12US1 modeling grid domain. Note:The datasets are on a Google Drive. The metadata associated with this DOI contain the link to the Google Drive folder and instructions for downloading the data. These files can be used with CMAQ-ISAMv5.3 to track state- or region-specific emissions. See Chapter 11 and Appendix B.4 in the CMAQ User's Guide for further information on how to use the ISAM control file with GRIDMASK files. The files can also be used for state or region-specific scaling of emissions using the CMAQv5.3 DESID module. See the DESID Tutorial and Appendix B.4 in the CMAQ User's Guide for further information on how to use the Emission Control File to scale emissions in predetermined geographical areas. File Location and Download Instructions: Link to GRIDMASK files Link to README text file with information on how these files were created File Format: The grid mask are stored as netcdf formatted files using I/O API data structures (https://www.cmascenter.org/ioapi/). Information on the model projection and grid structure is contained in the header information of the netcdf file. The output files can be opened and manipulated using I/O API utilities (e.g. M3XTRACT, M3WNDW) or other software programs that can read and write netcdf formatted files (e.g. Fortran, R, Python). File descriptions These GRIDMASK files can be used with the 12US1 modeling grid domain (grid origin x = -2556000 m, y = -1728000 m; N columns = 459, N rows = 299). GRIDMASK_STATES_12US1.nc - This file containes 49 variables for the 48 states in the conterminous U.S. plus DC. Each state variable (e.g., AL, AZ, AR, etc.) is a 2D array (299 x 459) providing the fractional area of each grid cell that falls within that state. GRIDMASK_CLIMATE_REGIONS_12US1.nc - This file containes 9 variables for 9 NOAA climate regions based on the Karl and Koss (1984) definition of climate regions. Each climate region variable (e.g., CLIMATE_REGION_1, CLIMATE_REGION_2, etc.) is a 2D array (299 x 459) providing the fractional area of each grid cell that falls within that climate region. NOAA Climate regions: CLIMATE_REGION_1: Northwest (OR, WA, ID) CLIMATE_REGION_2: West (CA, NV) CLIMATE_REGION_3: West North Central (MT, WY, ND, SD, NE) CLIMATE_REGION_4: Southwest (UT, AZ, NM, CO) CLIMATE_REGION_5: South (KS, OK, TX, LA, AR, MS) CLIMATE_REGION_6: Central (MO, IL, IN, KY, TN, OH, WV) CLIMATE_REGION_7: East North Central (MN, IA, WI, MI) CLIMATE_REGION_8: Northeast (MD, DE, NJ, PA, NY, CT, RI, MA, VT, NH, ME) + Washington, D.C.* CLIMATE_REGION_9: Southeast (VA, NC, SC, GA, AL, GA) *Note that Washington, D.C. is not included in any of the climate regions on the website but was included with the “Northeast” region for the generation of this GRIDMASK file.

This dataset was produced with funding from the European Space Agency (ESA) Climate Change Initiative (CCI) Plus Soil Moisture Project (CCN 3 to ESRIN Contract No: 4000126684/19/I-NB "ESA CCI+ Phase 1 New R&D on CCI ECVS Soil Moisture"). Project website: https://climate.esa.int/en/projects/soil-moisture/

This dataset contains information on the Surface Soil Moisture (SM) content derived from satellite observations in the microwave domain.

Dataset paper (public preprint)

A description of this dataset, including the methodology and validation results, is available at:

Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: An independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data Discuss. [preprint], https://doi.org/10.5194/essd-2024-610, in review, 2025.

Abstract

ESA CCI Soil Moisture is a multi-satellite climate data record that consists of harmonized, daily observations coming from 19 satellites (as of v09.1) operating in the microwave domain. The wealth of satellite information, particularly over the last decade, facilitates the creation of a data record with the highest possible data consistency and coverage.
However, data gaps are still found in the record. This is particularly notable in earlier periods when a limited number of satellites were in operation, but can also arise from various retrieval issues, such as frozen soils, dense vegetation, and radio frequency interference (RFI). These data gaps present a challenge for many users, as they have the potential to obscure relevant events within a study area or are incompatible with (machine learning) software that often relies on gap-free inputs.
Since the requirement of a gap-free ESA CCI SM product was identified, various studies have demonstrated the suitability of different statistical methods to achieve this goal. A fundamental feature of such gap-filling method is to rely only on the original observational record, without need for ancillary variable or model-based information. Due to the intrinsic challenge, there was until present no global, long-term univariate gap-filled product available. In this version of the record, data gaps due to missing satellite overpasses and invalid measurements are filled using the Discrete Cosine Transform (DCT) Penalized Least Squares (PLS) algorithm (Garcia, 2010). A linear interpolation is applied over periods of (potentially) frozen soils with little to no variability in (frozen) soil moisture content. Uncertainty estimates are based on models calibrated in experiments to fill satellite-like gaps introduced to GLDAS Noah reanalysis soil moisture (Rodell et al., 2004), and consider the gap size and local vegetation conditions as parameters that affect the gapfilling performance.

Summary

• Gap-filled global estimates of volumetric surface soil moisture from 1991-2023 at 0.25° sampling
• Fields of application (partial): climate variability and change, land-atmosphere interactions, global biogeochemical cycles and ecology, hydrological and land surface modelling, drought applications, and meteorology
• Method: Modified version of DCT-PLS (Garcia, 2010) interpolation/smoothing algorithm, linear interpolation over periods of frozen soils. Uncertainty estimates are provided for all data points.
• More information: See Preimesberger et al. (2025) and https://doi.org/10.5281/zenodo.8320869" target="_blank" rel="noopener">ESA CCI SM Algorithm Theoretical Baseline Document [Chapter 7.2.9] (Dorigo et al., 2023)

Programmatic Download

You can use command line tools such as wget or curl to download (and extract) data for multiple years. The following command will download and extract the complete data set to the local directory ~/Download on Linux or macOS systems.

#!/bin/bash

# Set download directory
DOWNLOAD_DIR=~/Downloads

base_url="https://researchdata.tuwien.at/records/3fcxr-cde10/files"

# Loop through years 1991 to 2023 and download & extract data
for year in {1991..2023}; do
  echo "Downloading $year.zip..."
  wget -q -P "$DOWNLOAD_DIR" "$base_url/$year.zip"
  unzip -o "$DOWNLOAD_DIR/$year.zip" -d $DOWNLOAD_DIR
  rm "$DOWNLOAD_DIR/$year.zip"
done

Data details

The dataset provides global daily estimates for the 1991-2023 period at 0.25° (~25 km) horizontal grid resolution. Daily images are grouped by year (YYYY), each subdirectory containing one netCDF image file for a specific day (DD), month (MM) in a 2-dimensional (longitude, latitude) grid system (CRS: WGS84). The file name has the following convention:

ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED_GAPFILLED-YYYYMMDD000000-fv09.1r1.nc

Data Variables

Each netCDF file contains 3 coordinate variables (WGS84 longitude, latitude and time stamp), as well as the following data variables:

• sm: (float) The Soil Moisture variable reflects estimates of daily average volumetric soil moisture content (m3/m3) in the soil surface layer (~0-5 cm) over a whole grid cell (0.25 degree).
• sm_uncertainty: (float) The Soil Moisture Uncertainty variable reflects the uncertainty (random error) of the original satellite observations and of the predictions used to fill observation data gaps.
• sm_anomaly: Soil moisture anomalies (reference period 1991-2020) derived from the gap-filled values (`sm`)
• sm_smoothed: Contains DCT-PLS predictions used to fill data gaps in the original soil moisture field. These values are also available when an observation was initially available (compare `gapmask`). In this case, they provided a smoothed version of the original data.
• gapmask: (0 | 1) Indicates grid cells where a satellite observation is available (0), and where the interpolated value is used (1) in the 'sm' field.
• frozenmask: (0 | 1) Indicates grid cells where ERA5 soil temperature is <0 °C. In this case, a linear interpolation over time is applied.

Additional information for each variable is given in the netCDF attributes.

Version Changelog

Changes in v9.1r1 (previous version was v09.1):

• This version uses a novel uncertainty estimation scheme as described in Preimesberger et al. (2025).

Software to open netCDF files

These data can be read by any software that supports Climate and Forecast (CF) conform metadata standards for netCDF files, such as:

• https://github.com/pydata/xarray" target="_blank" rel="noopener">Xarray (python)
• https://unidata.github.io/netcdf4-python/" target="_blank" rel="noopener">netCDF4 (python)
• https://github.com/TUW-GEO/esa_cci_sm">esa_cci_sm (python)
• Similar tools exists for other programming languages (Matlab, R, etc.)
• Software packages and GIS tools can open netCDF files, e.g. CDO, NCO, QGIS, ArCGIS
• You can also use the GUI software Panoply to view the contents of each file

References

• Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: An independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data Discuss. [preprint], https://doi.org/10.5194/essd-2024-610, in review, 2025.
• Dorigo, W., Preimesberger, W., Stradiotti, P., Kidd, R., van der Schalie, R., van der Vliet, M., Rodriguez-Fernandez, N., Madelon, R., & Baghdadi, N. (2023). ESA Climate Change Initiative Plus - Soil Moisture Algorithm Theoretical Baseline Document (ATBD) Supporting Product Version 08.1 (version 1.1). Zenodo. https://doi.org/10.5281/zenodo.8320869
• Garcia, D., 2010. Robust smoothing of gridded data in one and higher dimensions with missing values. Computational Statistics & Data Analysis, 54(4), pp.1167-1178. Available at: https://doi.org/10.1016/j.csda.2009.09.020
• Rodell, M., Houser, P. R., Jambor, U., Gottschalck, J., Mitchell, K., Meng, C.-J., Arsenault, K., Cosgrove, B., Radakovich, J., Bosilovich, M., Entin, J. K., Walker, J. P., Lohmann, D., and Toll, D.: The Global Land Data Assimilation System, Bulletin of the American Meteorological Society, 85, 381 – 394, https://doi.org/10.1175/BAMS-85-3-381, 2004.

Related Records

The following records are all part of the Soil Moisture Climate Data Records from satellites community

1

ESA CCI SM MODELFREE Surface Soil Moisture Record

<a href="https://doi.org/10.48436/svr1r-27j77" target="_blank"

The twin satellites of the Gravity Recovery and Climate Experiment (GRACE), launched in March of 2002, are making detailed monthly measurements of Earth's gravity field changes. These observations can detect regional mass changes of Earth's water reservoirs over land, ice and oceans. GRACE measures gravity variations by relating it to the distance variations between the two satellites, which fly in the same orbit, separated by about 240 km at an altitude of ~450 km. The monthly land mass grids contain terrestrial water storage anomalies (in aquifers, river basins, etc.) from GRACE time-variable gravity data relative to a time-mean. The storage anomalies are given in 'equivalent water thickness' (in NetCDF format). The time coverage for the monthly grids are determined by GRACE months. For the list of GRACE month dates visit http://grace.jpl.nasa.gov/data/grace-months/ . For information please visit http://grace.jpl.nasa.gov/data/get-data/monthly-mass-grids-land/ .

HRES is a high-resolution (100m) synthetic bed elevation terrain for the whole Antarctic continent. The synthetic bed surface preserves topographic roughness characteristics of airborne and ground-based ice-penetrating radar data from the Bedmap1 compilation and the ICECAP consortium. Broad-scale features of the Antarctic landscape are incorporated from a lowpass filter of the Bedmap2 bed elevation data. The data are available in NetCDF classic format on a 100m resolution grid in a Polar Stereographic Projection (Central Meridian 0 degrees, Standard Parallel 71 degrees S) with respect to the WGS84 geoid. The 100m grid is 66661 rows by 66661 columns, where the corner of the lower left cell is located at a polar stereographic easting and northing of -3333000 m and -3333000 m, respectively. The value for missing data is -9999.

A minor update was made to the dataset on 2021-12-23 to correct an error in the data.

The issue was that while the x and y dimensions were defined in the netcdf file (i.e. how many points in each dimension) the corresponding variables defining the actual locations were not defined. Therefore, when the file was loaded, the software was defaulting to using the index count to be the coordinate location, which would look like 1m data spacing

To fix this, the coordinate variables have been defined, and also the order the data was written in to correspond to what viewing programs such as ferret expect has been swapped.

The actual post-processing on the original data (hres.nc) was

gdal_translate -of netCDF -co WRITE_BOTTOMUP=NO NETCDF:"hres.nc":bed_elevation hres_temp_1.nc

ncpdq -a -y,x hres_temp_1.nc hres_temp_2.nc ncap2 -S hres_nco.txt hres_temp_2.nc hres_revised.nc

Where the file hres_nco.txt contains the following 8 lines y=array(-3331000f,100,$y); x=array(-3331000f,100,$x); y@long_name="Northing"; y@Standard_name="Northing"; y@units="m"; x@long_name="Easting"; x@Standard_name="Easting"; x@units="m";

The second processing step (ncpdq) requires a system with a bit over 40GB of memory.

This set of two zip files contains netcdf output files simulated with the WOMAPP model published as:

• van Oort, P.A.J., Verhagen, A., van der Werf, A.K., 2023. Can seaweeds feed the world? Modelling world offshore seaweed production potential. Ecological modelling

The two zip files contain model output for optimistic and pessimistic scenarios, referring to literature based data on seaweed nutrient requirements:

• in the optimistic scenario:
  • cold temperature seaweed species group is NOT sensitive to high waves
  • intermediate temperatures and warm temperature seaweed species groups: lower nutrient requirements
• in the pessimistic scenario:
  • cold temperature seaweed species group is sensitive to high waves
  • intermediate temperatures and warm temperature seaweed species groups: higher nutrient requirements

Within each zip file, the following netcdf files can be found:

Filename	Contents
WOMAPP_legendmonths.txt	QGIS legend file with colourscheme for number of suitable months (range 0-12)
WOMAPP_legendsuitability.txt	QGIS legend file with colourscheme for site suitability (range 0.0-1.0)
WorldEEZY_MonthSuitabilityAll_Cold.nc	Netcdf file for the world, for the cold temperature seaweed species group. Showing 7 simulated biophysical suitabilities (for all 6 environmental variables & overall suitability; range 0.0-1.0), for 12 months. Suitabilities outside the EEZ are all shown as zero (0.0). No data is shown as zero, which in some coastal spots looks a bit odd. When zooming in and finding a pixel with value zero close to the coast and with neighbouring pixels looking very suitable, then that pixel is probably a NO DATA pixel. Our apologies for this.
WorldEEZY_MonthSuitabilityAll_Intermediate.nc	Netcdf file for the world, for the intermediate temperature seaweed species group. Showing 7 simulated biophysical suitabilities (for all 6 environmental variables & overall suitability; range 0.0-1.0), for 12 months. Suitabilities outside the EEZ are all shown as zero (0.0). No data is shown as zero, which in some coastal spots looks a bit odd. When zooming in and finding a pixel with value zero close to the coast and with neighbouring pixels looking very suitable, then that pixel is probably a NO DATA pixel. Our apologies for this.
WorldEEZY_MonthSuitabilityAll_Warm.nc	Netcdf file for the world, for the warm temperature seaweed species group. Showing 7 simulated biophysical suitabilities (for all 6 environmental variables & overall suitability; range 0.0-1.0), for 12 months. Suitabilities outside the EEZ are all shown as zero (0.0). No data is shown as zero, which in some coastal spots looks a bit odd. When zooming in and finding a pixel with value zero close to the coast and with neighbouring pixels looking very suitable, then that pixel is probably a NO DATA pixel. Our apologies for this.
WorldEEZY_MonthSuitabilityGt_Cold.nc	Netcdf file for the world, for the cold temperature seaweed species group. Showing simulated number of suitable months (range 0-12) at 4 thresholds for overall suitability thresholds (0.5, 0.6, 0.7, 0.8). Outside the EEZ number of suitable months is set to zero (0). No data is shown as zero, which in some coastal spots looks a bit odd. When zooming in and finding a pixel with value zero close to the coast and with neighbouring pixels looking very suitable, then that pixel is probably a NO DATA pixel. Our apologies for this.
WorldEEZY_MonthSuitabilityGt_Intermediate.nc	Netcdf file for the world, for the intermediate temperature seaweed species group. Showing simulated number of suitable months (range 0-12) at 4 thresholds for overall suitability thresholds (0.5, 0.6, 0.7, 0.8). Outside the EEZ number of suitable months is set to zero (0). No data is shown as zero, which in some coastal spots looks a bit odd. When zooming in and finding a pixel with value zero close to the coast and with neighbouring pixels looking very suitable, then that pixel is probably a NO DATA pixel. Our apologies for this.
WorldEEZY_MonthSuitabilityGt_Warm.nc	Netcdf file for the world, for the warm temperature seaweed species group. Showing simulated number of suitable months (range 0-12) at 4 thresholds for overall suitability thresholds (0.5, 0.6, 0.7, 0.8). Outside the EEZ number of suitable months is set to zero (0). No data is shown as zero, which in some coastal spots looks a bit odd. When zooming in and finding a pixel with value zero close to the coast and with neighbouring pixels looking very suitable, then that pixel is probably a NO DATA pixel. Our apologies for this.
WorldEEZY_MonthSuitabilityOverall_Cold.nc	Netcdf file for the world, for the cold temperature seaweed species group. Showing for each of 12 months simulated overall suitability (range 0.0-1.0). Values outside the EEZ set to NULL
WorldEEZY_MonthSuitabilityOverall_Intermediate.nc	Netcdf file for the world, for the intermediate temperature seaweed species group. Showing for each of 12 months simulated overall suitability (range 0.0-1.0). Values outside the EEZ set to NULL
WorldEEZY_MonthSuitabilityOverall_Warm.nc	Netcdf file for the world, for the warm temperature seaweed species group. Showing for each of 12 months simulated overall suitability (range 0.0-1.0). Values outside the EEZ set to NULL
WorldY_MonthSuitabilityAll_Cold.nc	Netcdf file for the world, for the cold temperature seaweed species group. Showing 7 simulated biophysical suitabilities (for all 6 environmental variables & overall suitability; range 0.0-1.0), for 12 months.
WorldY_MonthSuitabilityAll_Intermediate.nc	Netcdf file for the world, for the intermediate temperature seaweed species group. Showing 7 simulated biophysical suitabilities (for all 6 environmental variables & overall suitability; range 0.0-1.0), for 12 months.
WorldY_MonthSuitabilityAll_Warm.nc	Netcdf file for the world, for the warm temperature seaweed species group. Showing 7 simulated biophysical suitabilities (for all 6 environmental variables & overall suitability; range 0.0-1.0), for 12 months.
WorldY_MonthSuitabilityGt_Cold.nc	Netcdf file for the world, for the cold temperature seaweed species group. Showing simulated number of suitable months (range 0-12) at 4 thresholds for overall suitability thresholds (0.5, 0.6, 0.7, 0.8).
WorldY_MonthSuitabilityGt_Intermediate.nc	Netcdf file for the world, for the intermediate temperature seaweed species group. Showing simulated number of suitable months (range 0-12) at 4 thresholds for overall suitability thresholds (0.5, 0.6, 0.7, 0.8).
WorldY_MonthSuitabilityGt_Warm.nc	Netcdf file for the world, for the warm temperature seaweed species group. Showing simulated number of suitable months (range 0-12) at 4 thresholds for overall suitability thresholds (0.5, 0.6, 0.7, 0.8).
WorldY_MonthSuitabilityOverall_Cold.nc	Netcdf file for the world, for the cold temperature seaweed species group. Showing for each of 12 months simulated overall suitability (range 0.0-1.0).
WorldY_MonthSuitabilityOverall_Intermediate.nc	Netcdf file for the world, for the intermediate temperature seaweed species group. Showing for each of 12 months simulated overall suitability (range 0.0-1.0).
WorldY_MonthSuitabilityOverall_Warm.nc	Netcdf file for the world, for the warm temperature seaweed species group. Showing for each of 12 months simulated overall suitability (range 0.0-1.0).

The research is important for the Great Barrier Reef (GBR) water quality management. The data was collected for the quantification of the contribution of Trichodesmium to the nitrogen budget of the GBR. Linux operating system, C compiler and NETCDF library were used to build the modified EMS applications on AIMS HPC. The EMS version used is 1.2.1. The modified EMS was derived from the eReefs model (https://ereefs.org.au/ereefs) and the model descriptions are found in Baird et al. (2020). Methods for collecting the data include the following:

Hydrodynamic model forcing available in https://research.csiro.au/ereefs/models/models-about/models-hydrodynamics/;

Biogeochemical (BGC) model forcing (Simulated hydrodynamic model output, regional wave model data, 2019 catchment conditions of nutrient and sediment loads available in https://svnserv.csiro.au/svn/CEM/projects/eReefs/model/gbr4_bgc_hindcast/gbr4_H2p0_B3p2_Cb/); Initialisation file: GBR4 BGC 3p1 initialisation data.

The 4km resolution grid of the EMS was run on AIMS HPC from 1/12/2010 to 30/11/2012 and the data was collected on 17/02/2022. Software-specific information needed to interpret the data are R Software version 3.5.1, GNU Compiler Collection (GCC) version 6.1.0, network Common Data Form (NetCDF-cxx) version 4.2.1, Open Message Passing Interface (OpenMPI-gcc) version 1.10.2 and NetCDF Operators (NCO) version 4.5.5. R scripts for post-processing simulated data are available in https://github.com/Chinenyeani1986/Trichodesmium-N-budget.

Daily averaged fractional ice cover in percentage, processed from passive microwave satellite data (SSMIS) over the polar regions. Sea ice concentration and its uncertainties are calculated from swath observations, and averaged and gridded to the daily fields. Better than using this archived NRT sea ice concentration product, please use the reprocessed sea ice concentration data record v2.0 (EO:EUM:DAT:MULT:OSI-450).

Retrieving multi-temporal max/min temperature data for specific locations from gridded nc files. The data are written in the CSV format.

This is the simulation output of the submitted manuscript "Between Broadening and Narrowing: How Mixing Affects the Width of the Droplet Size Distribution" to the Journal of Geophysical Research: Atmospheres.

data.tar file includes model output, including all variables used in the paper. The description of each variable is indicated in the nc file header. Each nc file is an output of a single time step in the whole model domain, where the simulation timestep is indicated as (time) at the end of each file (e.g., 000001440). The model timestep (dt) is 0.5s.

Files named SCMS_polluted_np120_nz120_history60step_dt0.5_ractstt_200_(time)_mod.nc are the results from simulation with a linear eddy model (LEM) and SCMS_polluted_nosgs_np120_nz120_history60step_dt0.5_ractstt_200_(time)_mod.nc are the results from the simulation without a LEM.

Overview

This dataset is the repository for the following paper submitted to Data in Brief:

Kempf, M. A dataset to model Levantine landcover and land-use change connected to climate change, the Arab Spring and COVID-19. Data in Brief (submitted: December 2023).

The Data in Brief article contains the supplement information and is the related data paper to:

Kempf, M. Climate change, the Arab Spring, and COVID-19 - Impacts on landcover transformations in the Levant. Journal of Arid Environments (revision submitted: December 2023).

Description/abstract

The Levant region is highly vulnerable to climate change, experiencing prolonged heat waves that have led to societal crises and population displacement. Since 2010, the area has been marked by socio-political turmoil, including the Syrian civil war and currently the escalation of the so-called Israeli-Palestinian Conflict, which strained neighbouring countries like Jordan due to the influx of Syrian refugees and increases population vulnerability to governmental decision-making. Jordan, in particular, has seen rapid population growth and significant changes in land-use and infrastructure, leading to over-exploitation of the landscape through irrigation and construction. This dataset uses climate data, satellite imagery, and land cover information to illustrate the substantial increase in construction activity and highlights the intricate relationship between climate change predictions and current socio-political developments in the Levant.

Folder structure

The main folder after download contains all data, in which the following subfolders are stored are stored as zipped files:

“code” stores the above described 9 code chunks to read, extract, process, analyse, and visualize the data.

“MODIS_merged” contains the 16-days, 250 m resolution NDVI imagery merged from three tiles (h20v05, h21v05, h21v06) and cropped to the study area, n=510, covering January 2001 to December 2022 and including January and February 2023.

“mask” contains a single shapefile, which is the merged product of administrative boundaries, including Jordan, Lebanon, Israel, Syria, and Palestine (“MERGED_LEVANT.shp”).

“yield_productivity” contains .csv files of yield information for all countries listed above.

“population” contains two files with the same name but different format. The .csv file is for processing and plotting in R. The .ods file is for enhanced visualization of population dynamics in the Levant (Socio_cultural_political_development_database_FAO2023.ods).

“GLDAS” stores the raw data of the NASA Global Land Data Assimilation System datasets that can be read, extracted (variable name), and processed using code “8_GLDAS_read_extract_trend” from the respective folder. One folder contains data from 1975-2022 and a second the additional January and February 2023 data.

“built_up” contains the landcover and built-up change data from 1975 to 2022. This folder is subdivided into two subfolder which contain the raw data and the already processed data. “raw_data” contains the unprocessed datasets and “derived_data” stores the cropped built_up datasets at 5 year intervals, e.g., “Levant_built_up_1975.tif”.

Code structure

1_MODIS_NDVI_hdf_file_extraction.R

This is the first code chunk that refers to the extraction of MODIS data from .hdf file format. The following packages must be installed and the raw data must be downloaded using a simple mass downloader, e.g., from google chrome. Packages: terra. Download MODIS data from after registration from: https://lpdaac.usgs.gov/products/mod13q1v061/ or https://search.earthdata.nasa.gov/search (MODIS/Terra Vegetation Indices 16-Day L3 Global 250m SIN Grid V061, last accessed, 09th of October 2023). The code reads a list of files, extracts the NDVI, and saves each file to a single .tif-file with the indication “NDVI”. Because the study area is quite large, we have to load three different (spatially) time series and merge them later. Note that the time series are temporally consistent.

2_MERGE_MODIS_tiles.R

In this code, we load and merge the three different stacks to produce large and consistent time series of NDVI imagery across the study area. We further use the package gtools to load the files in (1, 2, 3, 4, 5, 6, etc.). Here, we have three stacks from which we merge the first two (stack 1, stack 2) and store them. We then merge this stack with stack 3. We produce single files named NDVI_final_*consecutivenumber*.tif. Before saving the final output of single merged files, create a folder called “merged” and set the working directory to this folder, e.g., setwd("your directory_MODIS/merged").

3_CROP_MODIS_merged_tiles.R

Now we want to crop the derived MODIS tiles to our study area. We are using a mask, which is provided as .shp file in the repository, named "MERGED_LEVANT.shp". We load the merged .tif files and crop the stack with the vector. Saving to individual files, we name them “NDVI_merged_clip_*consecutivenumber*.tif. We now produced single cropped NDVI time series data from MODIS.
The repository provides the already clipped and merged NDVI datasets.

4_TREND_analysis_NDVI.R

Now, we want to perform trend analysis from the derived data. The data we load is tricky as it contains 16-days return period across a year for the period of 22 years. Growing season sums contain MAM (March-May), JJA (June-August), and SON (September-November). December is represented as a single file, which means that the period DJF (December-February) is represented by 5 images instead of 6. For the last DJF period (December 2022), the data from January and February 2023 can be added. The code selects the respective images from the stack, depending on which period is under consideration. From these stacks, individual annually resolved growing season sums are generated and the slope is calculated. We can then extract the p-values of the trend and characterize all values with high confidence level (0.05). Using the ggplot2 package and the melt function from reshape2 package, we can create a plot of the reclassified NDVI trends together with a local smoother (LOESS) of value 0.3.
To increase comparability and understand the amplitude of the trends, z-scores were calculated and plotted, which show the deviation of the values from the mean. This has been done for the NDVI values as well as the GLDAS climate variables as a normalization technique.

5_BUILT_UP_change_raster.R

Let us look at the landcover changes now. We are working with the terra package and get raster data from here: https://ghsl.jrc.ec.europa.eu/download.php?ds=bu (last accessed 03. March 2023, 100 m resolution, global coverage). Here, one can download the temporal coverage that is aimed for and reclassify it using the code after cropping to the individual study area. Here, I summed up different raster to characterize the built-up change in continuous values between 1975 and 2022.

6_POPULATION_numbers_plot.R

For this plot, one needs to load the .csv-file “Socio_cultural_political_development_database_FAO2023.csv” from the repository. The ggplot script provided produces the desired plot with all countries under consideration.

7_YIELD_plot.R

In this section, we are using the country productivity from the supplement in the repository “yield_productivity” (e.g., "Jordan_yield.csv". Each of the single country yield datasets is plotted in a ggplot and combined using the patchwork package in R.

8_GLDAS_read_extract_trend

The last code provides the basis for the trend analysis of the climate variables used in the paper. The raw data can be accessed https://disc.gsfc.nasa.gov/datasets?keywords=GLDAS%20Noah%20Land%20Surface%20Model%20L4%20monthly&page=1 (last accessed 9th of October 2023). The raw data comes in .nc file format and various variables can be extracted using the [“^a variable name”] command from the spatraster collection. Each time you run the code, this variable name must be adjusted to meet the requirements for the variables (see this link for abbreviations: https://disc.gsfc.nasa.gov/datasets/GLDAS_CLSM025_D_2.0/summary, last accessed 09th of October 2023; or the respective code chunk when reading a .nc file with the ncdf4 package in R) or run print(nc) from the code or use names(the spatraster collection).
Choosing one variable, the code uses the MERGED_LEVANT.shp mask from the repository to crop and mask the data to the outline of the study area.
From the processed data, trend analysis are conducted and z-scores were calculated following the code described above. However, annual trends require the frequency of the time series analysis to be set to value = 12. Regarding, e.g., rainfall, which is measured as annual sums and not means, the chunk r.sum=r.sum/12 has to be removed or set to r.sum=r.sum/1 to avoid calculating annual mean values (see other variables). Seasonal subset can be calculated as described in the code. Here, 3-month subsets were chosen for growing seasons, e.g. March-May (MAM), June-July (JJA), September-November (SON), and DJF (December-February, including Jan/Feb of the consecutive year).
From the data, mean values of 48 consecutive years are calculated and trend analysis are performed as describe above. In the same way, p-values are extracted and 95 % confidence level values are marked with dots on the raster plot. This analysis can be performed with a much longer time series, other variables, ad different spatial extent across the globe due to the availability of the GLDAS variables.

Single-beam bathymetry and magnetics data along with DGPS navigation data was collected as part of field activity L-5-85-NC in Northern California from 08/10/1985 to 08/31/1985, http://walrus.wr.usgs.gov/infobank/l/l585nc/html/l-5-85-nc.meta.html These data are reformatted from space-delimited ASCII text files located in the Coastal and Marine Geology Program (CMGP) InfoBank field activity catalog at http://walrus.wr.usgs.gov/infobank/l/l585nc/html/l-5-85-nc.bath.html and http://walrus.wr.usgs.gov/infobank/l/l585nc/html/l-5-85-nc.mag.html into MGD77T format provided by the NOAA's National Geophysical Data Center(NGDC). The MGD77T format includes a header (documentation) file (.h77t) and a data file (.m77t). More information regarding this format can be found in the publication listed in the Cross_reference section of this metadata file.

The TIGER/Line shapefiles and related database files (.dbf) are an extract of selected geographic and cartographic information from the U.S. Census Bureau's Master Address File / Topologically Integrated Geographic Encoding and Referencing (MAF/TIGER) Database (MTDB). The MTDB represents a seamless national file with no overlaps or gaps between parts, however, each TIGER/Line shapefile is designed to stand alone as an independent data set, or they can be combined to cover the entire nation. The Feature Names Relationship File (FEATNAMES.dbf) contains a record for each feature name and any attributes associated with it. Each feature name can be linked to the corresponding edges that make up that feature in the All Lines Shapefile (EDGES.shp), where applicable to the corresponding address range or ranges in the Address Ranges Relationship File (ADDR.dbf), or to both files. Although this file includes feature names for all linear features, not just road features, the primary purpose of this relationship file is to identify all street names associated with each address range. An edge can have several feature names; an address range located on an edge can be associated with one or any combination of the available feature names (an address range can be linked to multiple feature names). The address range is identified by the address range identifier (ARID) attribute, which can be used to link to the Address Ranges Relationship File (ADDR.dbf). The linear feature is identified by the linear feature identifier (LINEARID) attribute, which can be used to relate the address range back to the name attributes of the feature in the Feature Names Relationship File or to the feature record in the Primary Roads, Primary and Secondary Roads, or All Roads Shapefiles. The edge to which a feature name applies can be determined by linking the feature name record to the All Lines Shapefile (EDGES.shp) using the permanent edge identifier (TLID) attribute. The address range identifier(s) (ARID) for a specific linear feature can be found by using the linear feature identifier (LINEARID) from the Feature Names Relationship File (FEATNAMES.dbf) through the Address Range / Feature Name Relationship File (ADDRFN.dbf).

Single-beam bathymetry, gravity, and magnetic data along with DGPS navigation data was collected as part of field activity L-3-82-NC in Off San Mateo County, Northern California from 02/27/1982 to 03/01/1982, http://walrus.wr.usgs.gov/infobank/l/l382nc/html/l-3-82-nc.meta.html These data are reformatted from space-delimited ASCII text files located in the Coastal and Marine Geology Program (CMGP) InfoBank field activity catalog at http://walrus.wr.usgs.gov/infobank/l/l382nc/html/l-3-82-nc.bath.html, http://walrus.wr.usgs.gov/infobank/l/l382nc/html/l-3-82-nc.grav.html, and http://walrus.wr.usgs.gov/infobank/l/l382nc/html/l-3-82-nc.mag.html into MGD77T format provided by the NOAA's National Geophysical Data Center(NGDC). The MGD77T format includes a header (documentation) file (.h77t) and a data file (.m77t). More information regarding this format can be found in the publication listed in the Cross_reference section of this metadata file.

Airborne radiometric data are provided here as part of the data release "Airborne magnetic and radiometric survey, Virginia and North Carolina Fall Zone, 2021." This website includes the processed aeroradiometric data (gamma spectrometry for K, U and Th) in an ascii .csv file and geoTIFF images showing the total count values and potassium, thorium, and uranium concentrations. The spectral data are provided in a netcdf file that can be reviewed using any freely available viewer (such as HDFview) or loaded using software or programming languages with netcdf support, such as GSPy for Python (Foks, 2022). The contractor report is available on the parent page. References: Foks, N.L., James, S. R., and Minsely, B. J. 2022. GSPy: Geophysical Data Standard in Python. U.S. Geological Survey software release. doi:10.5066/P9XNQVGQ

Single-beam bathymetry data along with transit satellite navigation data was collected as part of field activity C-1-80-NC in Monterey Bay, Northern California from 05/21/1980 to 05/22/1980, http://walrus.wr.usgs.gov/infobank/c/c180nc/html/c-1-80-nc.meta.html The geophysical source was a Knudsen 12 kHz 320B/R echosounder. These data are reformatted from space-delimited ASCII text files located in the Coastal and Marine Geology Program (CMGP) InfoBank field activity catalog at http://walrus.wr.usgs.gov/infobank/c/c180nc/html/c-1-80-nc.bath.html into MGD77T format provided by the NOAA's National Geophysical Data Center(NGDC). The MGD77T format includes a header (documentation) file (.h77t) and a data file (.m77t). More information regarding this format can be found in the publication listed in the Cross_reference section of this metadata file.