Function name "cdf2mat" Please use this function to open MS-based chromatographic data from NETCDF (*.CDF) files. Resampling included for non-integer acquisition rates. Outputs nominal mass. Script optimized to process data from comprehensive two-dimensional gas chromatography coupled to mass spectrometry (GCxGC-MS). Updated to remove negative noise signal. INPUT file: Opens the netCDF like 'Sample01.CDF' rate_MS: Desired integer acquisition rate OUTPUT FullMS Full MS chromatogram (second order data tensor) axis_min Retention time axis in minutes axis_mz m/z axis in Daltons I/O: [TIC,FullMS,axis_min,axis_mz] = cdf2mat(file,rate_MS) Compiled with MATLAB R2021b (v.9.11.0.1809720). Requires the Signal Processing Toolbox (v.9.0). Based on netCDFload.m (Murphy, Wenig, Parcsi, Skov e Stuetz) e de iCDF_load (Skov e Bro 2008). K.R. Murphy, P. Wenig, G. Parcsi, T. Skov, R.M. Stuetz (in press) Characterizing odorous emissions using new software for identifying peaks in chemometric models of GC-MS datasets. Chem Intel Lab Sys. doi: 10.1016/j.chemolab.2012.07.006 Skov T and Bro R. (2008) Solving fundamental problems in chromatographic analysis, Analytical and Bioanalytical Chemistry, 390 (1): 281-285. doi: 10.1007/s00216-007-1618-z

This dataset contains information on the Surface Soil Moisture (SM) content derived from satellite observations in the microwave domain.

Dataset paper (public preprint)

A description of this dataset, including the methodology and validation results, is available at:

Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: An independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data Discuss. [preprint], https://doi.org/10.5194/essd-2024-610, in review, 2025.

Abstract

ESA CCI Soil Moisture is a multi-satellite climate data record that consists of harmonized, daily observations coming from 19 satellites (as of v09.1) operating in the microwave domain. The wealth of satellite information, particularly over the last decade, facilitates the creation of a data record with the highest possible data consistency and coverage.
However, data gaps are still found in the record. This is particularly notable in earlier periods when a limited number of satellites were in operation, but can also arise from various retrieval issues, such as frozen soils, dense vegetation, and radio frequency interference (RFI). These data gaps present a challenge for many users, as they have the potential to obscure relevant events within a study area or are incompatible with (machine learning) software that often relies on gap-free inputs.
Since the requirement of a gap-free ESA CCI SM product was identified, various studies have demonstrated the suitability of different statistical methods to achieve this goal. A fundamental feature of such gap-filling method is to rely only on the original observational record, without need for ancillary variable or model-based information. Due to the intrinsic challenge, there was until present no global, long-term univariate gap-filled product available. In this version of the record, data gaps due to missing satellite overpasses and invalid measurements are filled using the Discrete Cosine Transform (DCT) Penalized Least Squares (PLS) algorithm (Garcia, 2010). A linear interpolation is applied over periods of (potentially) frozen soils with little to no variability in (frozen) soil moisture content. Uncertainty estimates are based on models calibrated in experiments to fill satellite-like gaps introduced to GLDAS Noah reanalysis soil moisture (Rodell et al., 2004), and consider the gap size and local vegetation conditions as parameters that affect the gapfilling performance.

Summary

• Gap-filled global estimates of volumetric surface soil moisture from 1991-2023 at 0.25° sampling
• Fields of application (partial): climate variability and change, land-atmosphere interactions, global biogeochemical cycles and ecology, hydrological and land surface modelling, drought applications, and meteorology
• Method: Modified version of DCT-PLS (Garcia, 2010) interpolation/smoothing algorithm, linear interpolation over periods of frozen soils. Uncertainty estimates are provided for all data points.
• More information: See Preimesberger et al. (2025) and https://doi.org/10.5281/zenodo.8320869" target="_blank" rel="noopener">ESA CCI SM Algorithm Theoretical Baseline Document [Chapter 7.2.9] (Dorigo et al., 2023)

Programmatic Download

You can use command line tools such as wget or curl to download (and extract) data for multiple years. The following command will download and extract the complete data set to the local directory ~/Download on Linux or macOS systems.

#!/bin/bash

# Set download directory
DOWNLOAD_DIR=~/Downloads

base_url="https://researchdata.tuwien.at/records/3fcxr-cde10/files"

# Loop through years 1991 to 2023 and download & extract data
for year in {1991..2023}; do
  echo "Downloading $year.zip..."
  wget -q -P "$DOWNLOAD_DIR" "$base_url/$year.zip"
  unzip -o "$DOWNLOAD_DIR/$year.zip" -d $DOWNLOAD_DIR
  rm "$DOWNLOAD_DIR/$year.zip"
done

Data details

The dataset provides global daily estimates for the 1991-2023 period at 0.25° (~25 km) horizontal grid resolution. Daily images are grouped by year (YYYY), each subdirectory containing one netCDF image file for a specific day (DD), month (MM) in a 2-dimensional (longitude, latitude) grid system (CRS: WGS84). The file name has the following convention:

ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED_GAPFILLED-YYYYMMDD000000-fv09.1r1.nc

Data Variables

Each netCDF file contains 3 coordinate variables (WGS84 longitude, latitude and time stamp), as well as the following data variables:

• sm: (float) The Soil Moisture variable reflects estimates of daily average volumetric soil moisture content (m3/m3) in the soil surface layer (~0-5 cm) over a whole grid cell (0.25 degree).
• sm_uncertainty: (float) The Soil Moisture Uncertainty variable reflects the uncertainty (random error) of the original satellite observations and of the predictions used to fill observation data gaps.
• sm_anomaly: Soil moisture anomalies (reference period 1991-2020) derived from the gap-filled values (`sm`)
• sm_smoothed: Contains DCT-PLS predictions used to fill data gaps in the original soil moisture field. These values are also provided for cases where an observation was initially available (compare `gapmask`). In this case, they provided a smoothed version of the original data.
• gapmask: (0 | 1) Indicates grid cells where a satellite observation is available (1), and where the interpolated (smoothed) values are used instead (0) in the 'sm' field.
• frozenmask: (0 | 1) Indicates grid cells where ERA5 soil temperature is <0 °C. In this case, a linear interpolation over time is applied.

Additional information for each variable is given in the netCDF attributes.

Version Changelog

Changes in v9.1r1 (previous version was v09.1):

• This version uses a novel uncertainty estimation scheme as described in Preimesberger et al. (2025).

Software to open netCDF files

These data can be read by any software that supports Climate and Forecast (CF) conform metadata standards for netCDF files, such as:

• https://github.com/pydata/xarray" target="_blank" rel="noopener">Xarray (python)
• https://unidata.github.io/netcdf4-python/" target="_blank" rel="noopener">netCDF4 (python)
• https://github.com/TUW-GEO/esa_cci_sm">esa_cci_sm (python)
• Similar tools exists for other programming languages (Matlab, R, etc.)
• Software packages and GIS tools can open netCDF files, e.g. CDO, NCO, QGIS, ArCGIS
• You can also use the GUI software Panoply to view the contents of each file

References

• Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: An independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data Discuss. [preprint], https://doi.org/10.5194/essd-2024-610, in review, 2025.
• Dorigo, W., Preimesberger, W., Stradiotti, P., Kidd, R., van der Schalie, R., van der Vliet, M., Rodriguez-Fernandez, N., Madelon, R., & Baghdadi, N. (2023). ESA Climate Change Initiative Plus - Soil Moisture Algorithm Theoretical Baseline Document (ATBD) Supporting Product Version 08.1 (version 1.1). Zenodo. https://doi.org/10.5281/zenodo.8320869
• Garcia, D., 2010. Robust smoothing of gridded data in one and higher dimensions with missing values. Computational Statistics & Data Analysis, 54(4), pp.1167-1178. Available at: https://doi.org/10.1016/j.csda.2009.09.020
• Rodell, M., Houser, P. R., Jambor, U., Gottschalck, J., Mitchell, K., Meng, C.-J., Arsenault, K., Cosgrove, B., Radakovich, J., Bosilovich, M., Entin, J. K., Walker, J. P., Lohmann, D., and Toll, D.: The Global Land Data Assimilation System, Bulletin of the American Meteorological Society, 85, 381 – 394, https://doi.org/10.1175/BAMS-85-3-381, 2004.

Related Records

The following records are all part of the Soil Moisture Climate Data Records from satellites community

Netcdf data and Matlab processing scripts for the article:

Eabry, Holmes and Sen Gupta (2022): The impact of Indonesian Throughflow constrictions on eastern Pacific upwelling and water-mass transformation. Journal of Geophysical Research: Oceans. https://doi.org/10.1029/2022JC018509

Included are netcdf files with output from the ACCESS-OM2 1-degree ocean model averaged over years 500-600 of the spin-up simulation. CONTROL indicates the control simulation (realistic ITF topography), OPENITF indicates the Open ITF experiment and DIFF indicates difference files between the two. Please refer to the meta-data within the netcdf files for more information. Scripts to help with plotting standard variables are part of the COSIMA cookbook repository at https://github.com/COSIMA/cosima-recipes.

An example script Control_WMT_budget.m is provided to plot the control WMT budget and can be easily modified to plot the Open ITF or anomalous WMT budget. This script uses the Pacific masks found in mask.mat. The small tendency term is provided separately as dV_dt_nrho.mat.

Sea lice particle tracking data for a simulation of sea lice particles released from 20 farms in the Broughton Archipelago, Canada, from March 4th to July 20th, 2009. The output files are stored in netCDF format and so software that can read netCDF files is required to open the data files (R, MATLAB, and others). The associated R code was used to run the analysis in the manuscript. See README.txt file for details.

(I) DESCRIPTION:

· A global daily seamless 9-km Vegetation Optical Depth (VOD) product is generated through gap-filling and spatiotemporal fusion model. This daily products start from Jan 01, 2010 to Jul 31, 2021 (about 20GB memory after uncompressing all zip files).

· To further validate the effectiveness of these products, three verification ways are employed as follow: 1) Time series validation; 2) Simulated missing-region validation; And 3) Data comparison validation.

· It is important to note that the original data contain missing dates, and these corresponding gaps are also present in our dataset.

(II) DATA FORMATTING AND FILE NAMES

For the convenience of our readers, we have two formats of data available for download.

1) MAT file (Version v1)

Data from 2010 to 2021 are stored separately into folders for the corresponding years, with each folder containing daily .mat files. The naming convention for the data is “YYYYXXZZ,” where YYYY is the 4-digit year, XX is the 2-digit month, and ZZ is the 2-digit date. The geographic scope is global and the grid size is 4000*2000.

MATFILES (.mat): The folders with matfiles contain individual files for:

1. Vegetation Optical Depth: VOD_seamless_9km_ YYYYXXZZ.mat

2. Latitude/Longitude: VOD_9km_Coordinates.mat

2) NetCDF file (Version v2)

The year-by-year daily data from 2010 to 2021 are stored in the ‘.nc’ files for the corresponding years. The daily data within each year into one NetCDF file. The variable names are named as VOD_xxxxyydd, where xxxx represents the year, yy represents the month, and dd represents the day. The longitude variable is named “lon” with a dimension of 4000×1, and the latitude variable is named “lat” with a dimension of 2000×1.

It should be noted that these NetCDF files are saved using the netCDF4 library in Python, with the dimension order being (lat, lon). When reading these NetCDF files in MATLAB, the default data dimension order is (lon, lat). Therefore, it is necessary to transpose the variables to match the correct dimension order.