Testing files for the xesmf remapping package.

We implemented automated workflows using Jupyter notebooks for each state. The GIS processing, crucial for merging, extracting, and projecting GeoTIFF data, was performed using ArcPy—a Python package for geographic data analysis, conversion, and management within ArcGIS (Toms, 2015). After generating state-scale LES (large extent spatial) datasets in GeoTIFF format, we utilized the xarray and rioxarray Python packages to convert GeoTIFF to NetCDF. Xarray is a Python package to work with multi-dimensional arrays and rioxarray is rasterio xarray extension. Rasterio is a Python library to read and write GeoTIFF and other raster formats. Xarray facilitated data manipulation and metadata addition in the NetCDF file, while rioxarray was used to save GeoTIFF as NetCDF. These procedures resulted in the creation of three HydroShare resources (HS 3, HS 4 and HS 5) for sharing state-scale LES datasets. Notably, due to licensing constraints with ArcGIS Pro, a commercial GIS software, the Jupyter notebook development was undertaken on a Windows OS.

This dataset contains information on the Surface Soil Moisture (SM) content derived from satellite observations in the microwave domain.

Dataset Paper (Open Access)

A description of this dataset, including the methodology and validation results, is available at:

Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: an independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data, 17, 4305–4329, https://doi.org/10.5194/essd-17-4305-2025, 2025.

Abstract

ESA CCI Soil Moisture is a multi-satellite climate data record that consists of harmonized, daily observations coming from 19 satellites (as of v09.1) operating in the microwave domain. The wealth of satellite information, particularly over the last decade, facilitates the creation of a data record with the highest possible data consistency and coverage.
However, data gaps are still found in the record. This is particularly notable in earlier periods when a limited number of satellites were in operation, but can also arise from various retrieval issues, such as frozen soils, dense vegetation, and radio frequency interference (RFI). These data gaps present a challenge for many users, as they have the potential to obscure relevant events within a study area or are incompatible with (machine learning) software that often relies on gap-free inputs.
Since the requirement of a gap-free ESA CCI SM product was identified, various studies have demonstrated the suitability of different statistical methods to achieve this goal. A fundamental feature of such gap-filling method is to rely only on the original observational record, without need for ancillary variable or model-based information. Due to the intrinsic challenge, there was until present no global, long-term univariate gap-filled product available. In this version of the record, data gaps due to missing satellite overpasses and invalid measurements are filled using the Discrete Cosine Transform (DCT) Penalized Least Squares (PLS) algorithm (Garcia, 2010). A linear interpolation is applied over periods of (potentially) frozen soils with little to no variability in (frozen) soil moisture content. Uncertainty estimates are based on models calibrated in experiments to fill satellite-like gaps introduced to GLDAS Noah reanalysis soil moisture (Rodell et al., 2004), and consider the gap size and local vegetation conditions as parameters that affect the gapfilling performance.

Summary

• Gap-filled global estimates of volumetric surface soil moisture from 1991-2023 at 0.25° sampling
• Fields of application (partial): climate variability and change, land-atmosphere interactions, global biogeochemical cycles and ecology, hydrological and land surface modelling, drought applications, and meteorology
• Method: Modified version of DCT-PLS (Garcia, 2010) interpolation/smoothing algorithm, linear interpolation over periods of frozen soils. Uncertainty estimates are provided for all data points.
• More information: See Preimesberger et al. (2025) and https://doi.org/10.5281/zenodo.8320869" target="_blank" rel="noopener">ESA CCI SM Algorithm Theoretical Baseline Document [Chapter 7.2.9] (Dorigo et al., 2023)

Programmatic Download

You can use command line tools such as wget or curl to download (and extract) data for multiple years. The following command will download and extract the complete data set to the local directory ~/Download on Linux or macOS systems.

#!/bin/bash

# Set download directory
DOWNLOAD_DIR=~/Downloads

base_url="https://researchdata.tuwien.at/records/3fcxr-cde10/files"

# Loop through years 1991 to 2023 and download & extract data
for year in {1991..2023}; do
  echo "Downloading $year.zip..."
  wget -q -P "$DOWNLOAD_DIR" "$base_url/$year.zip"
  unzip -o "$DOWNLOAD_DIR/$year.zip" -d $DOWNLOAD_DIR
  rm "$DOWNLOAD_DIR/$year.zip"
done

Data details

The dataset provides global daily estimates for the 1991-2023 period at 0.25° (~25 km) horizontal grid resolution. Daily images are grouped by year (YYYY), each subdirectory containing one netCDF image file for a specific day (DD), month (MM) in a 2-dimensional (longitude, latitude) grid system (CRS: WGS84). The file name has the following convention:

ESACCI-SOILMOISTURE-L3S-SSMV-COMBINED_GAPFILLED-YYYYMMDD000000-fv09.1r1.nc

Data Variables

Each netCDF file contains 3 coordinate variables (WGS84 longitude, latitude and time stamp), as well as the following data variables:

• sm: (float) The Soil Moisture variable reflects estimates of daily average volumetric soil moisture content (m3/m3) in the soil surface layer (~0-5 cm) over a whole grid cell (0.25 degree).
• sm_uncertainty: (float) The Soil Moisture Uncertainty variable reflects the uncertainty (random error) of the original satellite observations and of the predictions used to fill observation data gaps.
• sm_anomaly: Soil moisture anomalies (reference period 1991-2020) derived from the gap-filled values (`sm`)
• sm_smoothed: Contains DCT-PLS predictions used to fill data gaps in the original soil moisture field. These values are also provided for cases where an observation was initially available (compare `gapmask`). In this case, they provided a smoothed version of the original data.
• gapmask: (0 | 1) Indicates grid cells where a satellite observation is available (1), and where the interpolated (smoothed) values are used instead (0) in the 'sm' field.
• frozenmask: (0 | 1) Indicates grid cells where ERA5 soil temperature is <0 °C. In this case, a linear interpolation over time is applied.

Additional information for each variable is given in the netCDF attributes.

Version Changelog

Changes in v9.1r1 (previous version was v09.1):

• This version uses a novel uncertainty estimation scheme as described in Preimesberger et al. (2025).

Software to open netCDF files

These data can be read by any software that supports Climate and Forecast (CF) conform metadata standards for netCDF files, such as:

• https://github.com/pydata/xarray" target="_blank" rel="noopener">Xarray (python)
• https://unidata.github.io/netcdf4-python/" target="_blank" rel="noopener">netCDF4 (python)
• https://github.com/TUW-GEO/esa_cci_sm">esa_cci_sm (python)
• Similar tools exists for other programming languages (Matlab, R, etc.)
• Software packages and GIS tools can open netCDF files, e.g. CDO, NCO, QGIS, ArCGIS
• You can also use the GUI software Panoply to view the contents of each file

References

• Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: an independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data, 17, 4305–4329, https://doi.org/10.5194/essd-17-4305-2025, 2025.
• Dorigo, W., Preimesberger, W., Stradiotti, P., Kidd, R., van der Schalie, R., van der Vliet, M., Rodriguez-Fernandez, N., Madelon, R., & Baghdadi, N. (2023). ESA Climate Change Initiative Plus - Soil Moisture Algorithm Theoretical Baseline Document (ATBD) Supporting Product Version 08.1 (version 1.1). Zenodo. https://doi.org/10.5281/zenodo.8320869
• Garcia, D., 2010. Robust smoothing of gridded data in one and higher dimensions with missing values. Computational Statistics & Data Analysis, 54(4), pp.1167-1178. Available at: https://doi.org/10.1016/j.csda.2009.09.020
• Rodell, M., Houser, P. R., Jambor, U., Gottschalck, J., Mitchell, K., Meng, C.-J., Arsenault, K., Cosgrove, B., Radakovich, J., Bosilovich, M., Entin, J. K., Walker, J. P., Lohmann, D., and Toll, D.: The Global Land Data Assimilation System, Bulletin of the American Meteorological Society, 85, 381 – 394, https://doi.org/10.1175/BAMS-85-3-381, 2004.

Related Records

The following records are all part of the ESA CCI Soil Moisture science data records community

This resource contains Jupyter Python notebooks which are intended to be used to learn about the U.S. National Water Model (NWM). These notebooks explore NWM forecasts in various ways. NWM Notebooks 1, 2, and 3, access NWM forecasts directly from the NOAA NOMADS file sharing system. Notebook 4 accesses NWM forecasts from Google Cloud Platform (GCP) storage in addition to NOMADS. A brief summary of what each notebook does is included below:

Notebook 1 (NWM1_Visualization) focuses on visualization. It includes functions for downloading and extracting time series forecasts for any of the 2.7 million stream reaches of the U.S. NWM. It also demonstrates ways to visualize forecasts using Python packages like matplotlib.

Notebook 2 (NWM2_Xarray) explores methods for slicing and dicing NWM NetCDF files using the python library, XArray.

Notebook 3 (NWM3_Subsetting) is focused on subsetting NWM forecasts and NetCDF files for specified reaches and exporting NWM forecast data to CSV files.

Notebook 4 (NWM4_Hydrotools) uses Hydrotools, a new suite of tools for evaluating NWM data, to retrieve NWM forecasts both from NOMADS and from Google Cloud Platform storage where older NWM forecasts are cached. This notebook also briefly covers visualizing, subsetting, and exporting forecasts retrieved with Hydrotools.

NOTE: Notebook 4 Requires a newer version of NumPy that is not available on the default CUAHSI JupyterHub instance. Please use the instance "HydroLearn - Intelligent Earth" and ensure to run !pip install hydrotools.nwm_client[gcp].

The notebooks are part of a NWM learning module on HydroLearn.org. When the associated learning module is complete, the link to it will be added here. It is recommended that these notebooks be opened through the CUAHSI JupyterHub App on Hydroshare. This can be done via the 'Open With' button at the top of this resource page.

These data are ocean profile data measured by profiling Argo S2A floats at a specific latitude, longitude, and date nominally from the surface to 2000 meters depth. Pressure, in situ temperature (ITS-90), and practical salinity are provided at 1-m increments through the water column. Argo data from Gulf of Mexico (GOM) LC1 (9 floats) and LC2 (12 floats) were delayed mode quality controlled and submitted to Global Data Assembly Centers (GDACs) in May 2020. All available profiles are planned to be revisited and evaluated in early 2021. Float no. 4903233 started showing drift in salinity at profile no. 77, and the salinity data will be carefully examined with a new adjustment in early 2021. _NCProperties=version=2,netcdf=4.6.3,hdf5=1.10.4 cdm_altitude_proxy=PRES cdm_data_type=Profile cdm_profile_variables=profile comment=free text contributor_email=devops@rpsgroup.com contributor_name=RPS contributor_role=editor contributor_role_vocabulary=https://vocab.nerc.ac.uk/collection/G04/current/ contributor_url=https://www.rpsgroup.com/ Conventions=CF-1.7, ACDD-1.3, IOOS-1.2, Argo-3.2, COARDS date_metadata_modified=2020-12-22T15:54:25Z Easternmost_Easting=-86.32756 featureType=Profile geospatial_bounds=POINT (-86.32756 26.2932) geospatial_bounds_crs=EPSG:4326 geospatial_lat_max=26.2932 geospatial_lat_min=26.2932 geospatial_lat_units=degrees_north geospatial_lon_max=-86.32756 geospatial_lon_min=-86.32756 geospatial_lon_units=degrees_east history=2020-04-15T22:00:52Z creation id=R4903232_060 infoUrl=http://www.argodatamgt.org/Documentation institution=GCOOS instrument=Argo instrument_vocabulary=GCMD Earth Science Keywords. Version 5.3.3 keywords_vocabulary=GCMD Science Keywords naming_authority=edu.tamucc.gulfhub Northernmost_Northing=26.2932 note_CHAR_variables=RPS METADATA ENHANCEMENT NOTE Variables of data type 'CHAR' have been altered by the xarray and netCDF4-python libraries to contain an extra dimension (often denoted as 'string1'). This is due to an underlying issue in the libraries: https://github.com/pydata/xarray/issues/1977. Upon examination, one will find the data has not been altered but only changed shape. We realize this is sub-optimal and apologize for any inconveniences this may cause. note_FillValue=RPS METADATA ENHANCEMENT NOTE Many variables in this dataset are of type 'char' and have a '_FillValue' attribute which is interpreted through NumPy as 'b', an empty byte string. This causes serialization issues. As a result, all variables of type 'char' with '_FillValue = b' have had the _FillValue attribute removed to avoid serialization conflicts. However, no data has been changed, so the _FillValue is still "b' '". platform=subsurface_float platform_name=Argo Float platform_vocabulary=IOOS Platform Vocabulary processing_level=Argo data are received via satellite transmission, decoded and assembled at national DACs. These DACs apply a set of automatic quality tests (RTQC) to the data, and quality flags are assigned accordingly. In the delayed-mode process (DMQC), data are subjected to visual examination and are re-flagged where necessary. For the float data affected by sensor drift, statistical tools and climatological comparisons are used to adjust the data for sensor drift when needed. For each float that has been processed in delayed-mode, the OWC method (Owens and Wong, 2009; Cabanes et al., 2016) is run with four different sets of spatial and temporal decorrelation scales and the latest available reference dataset. If the salinity adjustments obtained from the four runs all differ significantly from the existing adjustment, then the salinity data from the float are re-examined and a new adjustment is suggested if necessary. The usual practice is to examine the profiles in delayed-mode initially about 12 months after they are collected, and then revisit several times as more data from the floats are obtained (see details in Wong et al., 2020). program=Understanding Gulf Ocean Systems (UGOS) project=National Academy of Science Understanding Gulf Ocean Systems 'LC-Floats - Near Real-time Hydrography and Deep Velocity in the Loop Current System using Autonomous Profilers' Program references=http://www.argodatamgt.org/Documentation sea_name=Gulf of Mexico source=Argo float sourceUrl=(local files) Southernmost_Northing=26.2932 standard_name_vocabulary=CF Standard Name Table v67 subsetVariables=CYCLE_NUMBER, DIRECTION, DATA_MODE, time, JULD_QC, JULD_LOCATION, latitude, longitude, POSITION_QC, CONFIG_MISSION_NUMBER, PROFILE_PRES_QC, PROFILE_TEMP_QC, PROFILE_PSAL_QC time_coverage_duration=P0000-00-00T00:00:00 time_coverage_end=2020-04-10T19:39:59Z time_coverage_resolution=P0000-00-00T00:00:00 time_coverage_start=2020-04-10T19:39:59Z user_manual_version=3.2 Westernmost_Easting=-86.32756

This Zenodo repository contains all migration flow estimates associated with the paper "Deep learning four decades of human migration." Evaluation code, training data, trained neural networks, and smaller flow datasets are available in the main GitHub repository, which also provides detailed instructions on data sourcing. Due to file size limits, the larger datasets are archived here.

Data is available in both NetCDF (.nc) and CSV (.csv) formats. The NetCDF format is more compact and pre-indexed, making it suitable for large files. In Python, datasets can be opened as xarray.Dataset objects, enabling coordinate-based data selection.

Each dataset uses the following coordinate conventions:

• Year: 1990–2023
• Birth ISO: Country of birth (UN ISO3)
• Origin ISO: Country of origin (UN ISO3)
• Destination ISO: Destination country (UN ISO3)
• Country ISO: Used for net migration data (UN ISO3)

The following data files are provided:

• T.nc: Full table of flows disaggregated by country of birth. Dimensions: Year, Birth ISO, Origin ISO, Destination ISO
• flows.nc: Total origin-destination flows (equivalent to T summed over Birth ISO). Dimensions: Year, Origin ISO, Destination ISO
• net_migration.nc: Net migration data by country. Dimensions: Year, Country ISO
• stocks.nc: Stock estimates for each country pair. Dimensions: Year, Origin ISO (corresponding to Birth ISO), Destination ISO
• test_flows.nc: Flow estimates on a randomly selected set of test edges, used for model validation

Additionally, two CSV files are provided for convenience:

• mig_unilateral.csv: Unilateral migration estimates per country, comprising:
  • imm: Total immigration flows
  • emi: Total emigration flows
  • net: Net migration
  • imm_pop: Total immigrant population (non-native-born)
  • emi_pop: Total emigrant population (living abroad)
• mig_bilateral.csv: Bilateral flow data, comprising:
  • mig_prev: Total origin-destination flows
  • mig_brth: Total birth-destination flows, where Origin ISO reflects place of birth

Each dataset includes a mean variable (mean estimate) and a std variable (standard deviation of the estimate).

An ISO3 conversion table is also provided.

Phsyical Oceanographic Circulation Study _NCProperties=version=2,netcdf=4.6.3,hdf5=1.10.4 acknowledgement=N/A cdm_altitude_proxy=PRES cdm_data_type=Profile cdm_profile_variables=DIRECTION,JULD_QC,JULD_LOCATION,latitude,longitude,POSITION_QC,PROFILE_PRES_QC,PROFILE_PSAL_QC,PROFILE_TEMP_QC,PRES,PRES_QC,PRES_ADJUSTED,PRES_ADJUSTED_QC,PRES_ADJUSTED_ERROR,PSAL,PSAL_QC,PSAL_ADJUSTED,PSAL_ADJUSTED_QC,PSAL_ADJUSTED_ERROR,TEMP,TEMP_QC,TEMP_ADJUSTED,TEMP_ADJUSTED_QC,TEMP_ADJUSTED_ERROR,profile comment=N/A contributor_email=devops@rpsgroup.com contributor_name=RPS contributor_role=editor contributor_role_vocabulary=https://vocab.nerc.ac.uk/collection/G04/current/ contributor_url=rpsgroup.com Conventions=ACDD-1.3, CF-1.7, IOOS-1.2 date_metadata_modified=2021-03-15T14:51:43.046093 Easternmost_Easting=-86.85241 featureType=Profile geospatial_bounds=MULTIPOINT (-86.92479 27.44745, -86.94875 27.50054, -86.97515 27.55698, -86.98268 27.59141, -86.95361 27.75151, -87.08473 27.93702, -87.16455 27.8003 , -87.48748 27.69569, -87.79848 27.65302, -88.00196 27.50943, -87.9856 27.50101, -87.86047 27.63956, -88.06809 27.73081, -88.09864 27.78135, -88.10715 27.82397, -88.1108 27.85767, -88.10087 27.91051, -88.13338 27.86221, -88.16878 27.85377, -88.05664 28.00782, -88.02269 28.14189, -87.93753 28.32363, -87.77153 28.46238, -87.66765 28.34248, -87.59172 28.36801, -87.50798 28.56317, -87.38511 28.60202, -87.31172 28.60896, -87.28674 28.62204, -87.24137 28.4645 , -86.85241 28.17752, -87.06037 28.34397, -87.13829 28.3452 , -87.31952 28.54903, -87.38997 28.94832, -87.6393 29.06555, -87.41635 29.08024, -87.38668 28.99967, -87.47774 29.01179, -87.33341 28.79479, -87.45669 28.63872, -87.53495 28.50758, -87.54689 28.29854, -87.74877 28.45214, -87.79449 28.38046, -87.62084 28.30637, -87.65327 28.29676, -87.51157 28.37384, -87.52016 28.64711, -87.49763 28.90244, -87.6509 29.01665, -88.01583 28.92765, -88.35217 28.78962, -88.30744 28.64899, -88.17691 28.58762, -88.22921 28.35054, -88.66493 28.01616, -88.87602 27.89182, -89.26091 27.80431, -89.32015 27.70812, -89.37818 27.71823, -89.58081 27.68473, -89.54697 27.82638, -89.06157 28.11111, -88.64328 28.26838, -88.33775 28.23144, -88.58438 27.98143, -88.2644 26.93499, -88.62807 26.15466, -88.84813 26.56506, -89.15347 26.48345, -89.65883 26.93092, -89.35677 26.68739, -90.03865 26.61342, -90.55446 26.69557, -91.06189 26.41492, -91.43119 26.85679, -91.31301 26.97152, -90.96983 26.77915, -90.90907 26.79166, -90.99742 27.01004, -90.86617 26.89158, -90.83422 26.61472, -91.14603 26.23077, -92.33082 25.88828, -93.33612 26.01557, -94.64683 26.00732, -94.94638 26.10918, -95.11384 25.84752, -95.52623 24.59142, -95.94594 23.87184, -96.39403 23.6656 , -96.67977 23.76634, -96.92028 23.55518, -96.95649 23.3301 , -96.87699 23.22797) geospatial_bounds_crs=EPSG:4326 geospatial_bounds_vertical_crs=EPSG:5831 geospatial_lat_max=29.08024 geospatial_lat_min=23.22797 geospatial_lat_units=degrees_north geospatial_lon_max=-86.85241 geospatial_lon_min=-96.95649 geospatial_lon_units=degrees_east geospatial_vertical_positive=down history=_prof.nc and _meta.nc concatenated and enhanced metadata by RPS 2021-03-15T14:51:43.046075 id=4901476 infoUrl=https://gcoos.org institution=US Argo (US GDAC) instrument=US ARGO Profiler naming_authority=edu.tamucc.gulfhub Northernmost_Northing=29.08024 note_CHAR_variables=RPS METADATA ENHANCEMENT NOTE Variables of data type 'CHAR' have been altered by the xarray and netCDF4-python libraries to contain an extra dimension (often denoted as 'string1'). This is due to an underlying issue in the libraries: https://github.com/pydata/xarray/issues/1977. Upon examination, one will find the data has not been altered but only changed shape. We realize this is sub-optimal and apologize for any inconveniences this may cause. platform=subsurface_float platform_id=4901476 platform_name=US Argo APEX Float 4901476 platform_vocabulary=GCMD Keywords Version 8.7 processing_level=Data QA/QC performed by USARGO program=Deep Langrangian Observations in the Gulf of Mexico (funding: BOEM) project=Deep Circulation in the Gulf of Mexico: A Lagrangian Study references=https://espis.boem.gov/final%20reports/5471.pdf source=observation sourceUrl=(local files) Southernmost_Northing=23.22797 standard_name_vocabulary=CF Standard Name Table v72 time_coverage_duration=P0002-03-14T16:38:02 time_coverage_end=2015-11-21T22:46:01Z time_coverage_resolution=P0000-00-00T10:50:04 time_coverage_start=2013-08-07T06:07:59Z Westernmost_Easting=-96.95649

This is a netCDF. You can hence load it with xarray as a multi dimensional matrix with the date and the number of the station as the 2 dimensions. Latitude and longitude are attached as coordinates.

The notebook used to generate these data: https://www.kaggle.com/code/pierrel49/getting-wind-as-netcdf-from-se-meteonet

Abstract

Summary

• Global estimates of root-zone soil moisture from 01-1980 to 12-2024 at ~25 km spatial sampling based on the COMBINED product of ESA CCI SM v9.2.
• Method: Exponential filter model, calibrated with in situ measurements for 3 depth layers: 0-10, 10-40, 40-100 cm with uncertainty estimates. Additionally, one layer representing the average condition from 0-1 m depth is provided. See Pasik et al. (2023) for more details.
• Good agreement with independent reanalysis data (R ~0.6 [-] and ubRMSD <0.04 m³/m³), decreasing performance for deeper layers due to weaker coupling with surface SM.

Programmatic (bulk) download

You can use command-line tools such as wget or curl to download (and extract) data for multiple years. The following command will download and extract the complete data set to the local directory ~/Download on Linux or macOS systems.

#!/bin/bash

# Set download directory
DOWNLOAD_DIR=~/Downloads

base_url="https://researchdata.tuwien.at/records/tqrwj-t7r58/files"

# Loop through years 1980 to 2024 and download & extract data
for year in {1980..2024}; do
  echo "Downloading $year.zip..."
  wget -q -P "$DOWNLOAD_DIR" "$base_url/$year.zip"
  unzip -o "$DOWNLOAD_DIR/$year.zip" -d $DOWNLOAD_DIR
  rm "$DOWNLOAD_DIR/$year.zip"
done

Data details

Version Changelog

Changes in v9.2:

• The COMBINED product of v9.2 is used as input.
• The period was extended to 12-2024.

Software to open netCDF files

These data can be read by any software that supports Climate and Forecast (CF) conform metadata standards for netCDF files, such as:

• https://github.com/pydata/xarray" target="_blank" rel="noopener">Xarray (Python)
• https://unidata.github.io/netcdf4-python/" target="_blank" rel="noopener">netCDF4 (Python)
• https://github.com/TUW-GEO/esa_cci_sm" target="_blank" rel="noopener">esa_cci_sm (Python)
• Similar tools exist for other programming languages (Matlab, R, etc.)
• Software packages and GIS tools can open netCDF files, e.g. CDO, NCO, QGIS, ArcGIS
• You can also use the GUI software Panoply to view the contents of each file

Related Records

This record and all related records are part of the ESA CCI Soil Moisture science data records community.

'## This item has been replaced by the one which can be found at https://datashare.ed.ac.uk/handle/10283/4849 - https://doi.org/10.7488/ds/3843 ##' This archive contains the driving data and selected model outputs to accompany the manuscript: 'Resolving scale-variance in the carbon dynamics of fragmented, mixed-use landscapes estimated using Model-Data Fusion', submitted to Biogeosciences Discussions. The archive contains two zip files containing: (i) the observations and driving data assimilated into CARDAMOM; and (ii) a selection of model output, including the carbon (C) stocks for each DALEC pool, and a compilation of key C fluxes. Data and model output are stored as netcdf files. The xarray package (https://docs.xarray.dev/en/stable/index.html) provides a convenient starting point for using netcdf files within python environments. More details are provided in the document 'Milodowski_etal_dataset_description.pdf'

This data accompanies the article entitled Computational Insights into the Unfolding of a Destabilized Superoxide Dismutase 1 Mutant and published in Biology.

SOD1_WT_I35A_REST2.zip: The zip archive includes REST2 trajectories for the two SOD1 constructs and the two force fields investigated in the paper.

The trajectories are saved in the GROMACS XTC file format, separately for each temperature (i=0,...,23). Owing to the considerable trajectory sizes, only protein coordinates are reported, and the output frequency is reduced to 100 ps. The initial geometry (in the Gromos87 GRO format) after a short relaxation is provided for each REST2 simulation (conf_prot.gro). Furthermore, for each REST2 simulation, an xarray (http://xarray.pydata.org) dataset, saved in the netCDF file format, is included and contains the following observables: fraction of native contacts relative to the crystal structure, secondary-structure content (i.e., the fraction of protein residues found in an alpha-helix, beta-sheet, beta-bridge, or a turn), as well as the number of residues with the beta-sheet secondary structure per each beta-strand and beta-sheet of the SOD1 barrel.

IAGOS-CARIBIC WSM files collection (v2024.07.17)

Content

IAGOS-CARIBIC_WSM_files_collection_20240717.zip contains merged IAGOS-CARIBIC whole air sampler data (CARIBIC-1 and CARIBIC-2; <https://www.caribic-atmospheric.com/>). There is one netCDF file per IAGOS-CARIBIC flight. Files were generated from NASA Ames 1001. For detailed content information, see global and variable attributes. Global attribute `na_file_header_[x]` contains the original NASA Ames file header as an array of strings, with [x] being one of the source files.

Data Coverage

The data set covers 22 years of CARIBIC data from 1997 to 2020, flight numbers 8 to 591. There is no data available after 2020. Also, note that data isn't available for all flight numbers within the [1, 591] range.

Special note on CARIBIC-1 data

CARIBIC-1 data only contains a subset of the variables found in CARIBIC-2 data files. To distinguish those two campaigns, use the global attribute 'mission'.

File format

netCDF v4, created with xarray, <https://docs.xarray.dev/en/stable/>. Default variable encoding was used (no compression etc.).

Data availability

This dataset is also available via our THREDDS server at KIT, <https://thredds.atmohub.kit.edu/dataset/iagos-caribic-whole-air-sampler-data>.

Contact

Tanja Schuck, whole air sampling system PI,

Changelog

• `2024.07.17`: revise ozone data for flights 294 to 591
• `2024.01.22`: editorial changes, add Schuck et al. publications, data unchanged
• `2024.01.12`: initial upload

# ERA-NUTS (1980-2018)

This dataset contains a set of time-series of meteorological variables based on Copernicus Climate Change Service (C3S) ERA5 reanalysis. The data files can be downloaded from here while notebooks and other files can be found on the associated Github repository.

This data has been generated with the aim of providing hourly time-series of the meteorological variables commonly used for power system modelling and, more in general, studies on energy systems.

An example of the analysis that can be performed with ERA-NUTS is shown in this video.

Important: this dataset is still a work-in-progress, we will add more analysis and variables in the near-future. If you spot an error or something strange in the data please tell us sending an email or opening an Issue in the associated Github repository.

## Data
The time-series have hourly/daily/monthly frequency and are aggregated following the NUTS 2016 classification. NUTS (Nomenclature of Territorial Units for Statistics) is a European Union standard for referencing the subdivisions of countries (member states, candidate countries and EFTA countries).

This dataset contains NUTS0/1/2 time-series for the following variables obtained from the ERA5 reanalysis data (in brackets the name of the variable on the Copernicus Data Store and its unit measure):

- t2m: 2-meter temperature (`2m_temperature`, Celsius degrees)
- ssrd: Surface solar radiation (`surface_solar_radiation_downwards`, Watt per square meter)
- ssrdc: Surface solar radiation clear-sky (`surface_solar_radiation_downward_clear_sky`, Watt per square meter)
- ro: Runoff (`runoff`, millimeters)

There are also a set of derived variables:
- ws10: Wind speed at 10 meters (derived by `10m_u_component_of_wind` and `10m_v_component_of_wind`, meters per second)
- ws100: Wind speed at 100 meters (derived by `100m_u_component_of_wind` and `100m_v_component_of_wind`, meters per second)
- CS: Clear-Sky index (the ratio between the solar radiation and the solar radiation clear-sky)
- HDD/CDD: Heating/Cooling Degree days (derived by 2-meter temperature the EUROSTAT definition.

For each variable we have 350 599 hourly samples (from 01-01-1980 00:00:00 to 31-12-2019 23:00:00) for 34/115/309 regions (NUTS 0/1/2).

The data is provided in two formats:

- NetCDF version 4 (all the variables hourly and CDD/HDD daily). NOTE: the variables are stored as `int16` type using a `scale_factor` of 0.01 to minimise the size of the files.
- Comma Separated Value ("single index" format for all the variables and the time frequencies and "stacked" only for daily and monthly)

All the CSV files are stored in a zipped file for each variable.

## Methodology

The time-series have been generated using the following workflow:

1. The NetCDF files are downloaded from the Copernicus Data Store from the ERA5 hourly data on single levels from 1979 to present dataset
2. The data is read in R with the climate4r packages and aggregated using the function `/get_ts_from_shp` from panas. All the variables are aggregated at the NUTS boundaries using the average except for the runoff, which consists of the sum of all the grid points within the regional/national borders.
3. The derived variables (wind speed, CDD/HDD, clear-sky) are computed and all the CSV files are generated using R
4. The NetCDF are created using `xarray` in Python 3.7.

NOTE: air temperature, solar radiation, runoff and wind speed hourly data have been rounded with two decimal digits.

## Example notebooks

In the folder `notebooks` on the associated Github repository there are two Jupyter notebooks which shows how to deal effectively with the NetCDF data in `xarray` and how to visualise them in several ways by using matplotlib or the enlopy package.

There are currently two notebooks:

- exploring-ERA-NUTS: it shows how to open the NetCDF files (with Dask), how to manipulate and visualise them.
- ERA-NUTS-explore-with-widget: explorer interactively the datasets with [jupyter]() and ipywidgets.

The notebook `exploring-ERA-NUTS` is also available rendered as HTML.

## Additional files

In the folder `additional files`on the associated Github repository there is a map showing the spatial resolution of the ERA5 reanalysis and a CSV file specifying the number of grid points with respect to each NUTS0/1/2 region.

## License

This dataset is released under CC-BY-4.0 license.

Wave and sea surface temperature measurements collected by a Sofar Spotter buoy in 2023. The buoy was deployed on July 27, 2023 at 11:30 UTC northwest of Culebra Island, Puerto Rico, (18.3878 N, 65.3899 W) and recovered on Nov 5, 2023 at 12:45 UTC. Data are saved here in netCDF format, organized by month, and include directional wave statistics, GPS, and SST measurements at 30-minute intervals. Figures produced from these data are provided here as well. They include timeseries of wave height/period/direction and SST, GPS location, wave roses, and directional spectra. Additionally, raw CSV files from the Spotter's memory card can also be found below. NetCDF files can be read in python using the netCDF4 or Xarray packages, or through MATLAB using the "ncread()" command.

This dataset provides global daily estimates of Root-Zone Soil Moisture (RZSM) content at 0.25° spatial grid resolution, derived from gap-filled merged satellite observations of 14 passive satellites sensors operating in the microwave domain of the electromagnetic spectrum. Data is provided from January 1991 to December 2023.

This dataset was produced with funding from the European Space Agency (ESA) Climate Change Initiative (CCI) Plus Soil Moisture Project (CCN 3 to ESRIN Contract No: 4000126684/19/I-NB "ESA CCI+ Phase 1 New R&D on CCI ECVS Soil Moisture"). Project website: https://climate.esa.int/en/projects/soil-moisture/" target="_blank" rel="noopener">https://climate.esa.int/en/projects/soil-moisture/. Operational implementation is supported by the Copernicus Climate Change Service implemented by ECMWF through C3S2 312a/313c.

Studies using this dataset [preprint]

This dataset is used by Hirschi et al. (2025) to assess recent summer drought trends in Switzerland.

Hirschi, M., Michel, D., Schumacher, D. L., Preimesberger, W., and Seneviratne, S. I.: Recent summer soil moisture drying in Switzerland based on measurements from the SwissSMEX network, Earth Syst. Sci. Data Discuss. [preprint], https://doi.org/10.5194/essd-2025-416, in review, 2025.

Abstract

ESA CCI Soil Moisture is a multi-satellite climate data record that consists of harmonized, daily observations from various microwave satellite remote sensing sensors (Dorigo et al., 2017, 2024; Gruber et al., 2019). This version of the dataset uses the PASSIVE record as input, which contains only observations from passive (radiometer) measurements (scaling reference AMSR-E). The surface observations are gap-filled using a univariate interpolation algorithm (Preimesberger et al., 2025). The gap-filled passive observations serve as input for an exponential filter based method to assess soil moisture in different layers of the root-zone of soil (0-200 cm) following the approach by Pasik et al. (2023). The final gap-free root-zone soil moisture estimates based on passive surface input data are provided here at 4 separate depth layers (0-10, 10-40, 40-100, 100-200 cm) over the period 1991-2023.

Summary

• Gap-free root-zone soil moisture estimates from 1991-2023 at 0.25° spatial sampling from passive measurements
• Fields of application include: climate variability and change, land-atmosphere interactions, global biogeochemical cycles and ecology, hydrological and land surface modelling, drought applications, agriculture and meteorology
• More information: See Dorigo et al. (2017, 2024) and Gruber et al. (2019) for a description of the satellite base product and uncertainty estimates, Preimesberger et al. (2025) for the gap-filling, and Pasik et al. (2023) for the root-zone soil moisture and uncertainty propagation algorithm.

Programmatic Download

You can use command line tools such as wget or curl to download (and extract) data for multiple years. The following command will download and extract the complete data set to the local directory ~/Downloads on Linux or macOS systems.

#!/bin/bash

# Set download directory
DOWNLOAD_DIR=~/Downloads

base_url="https://researchdata.tuwien.ac.at/records/8dda4-xne96/files"

# Loop through years 1991 to 2023 and download & extract data
for year in {1991..2023}; do
  echo "Downloading $year.zip..."
  wget -q -P "$DOWNLOAD_DIR" "$base_url/$year.zip"
  unzip -o "$DOWNLOAD_DIR/$year.zip" -d $DOWNLOAD_DIR
  rm "$DOWNLOAD_DIR/$year.zip"
done

Data details

The dataset provides global daily estimates for the 1991-2023 period at 0.25° (~25 km) horizontal grid resolution. Daily images are grouped by year (YYYY), each subdirectory containing one netCDF image file for a specific day (DD), month (MM) in a 2-dimensional (longitude, latitude) grid system (CRS: WGS84). The file name has the following convention:

ESA_CCI_PASSIVERZSM-YYYYMMDD000000-fv09.1.nc

Data Variables

Each netCDF file contains 3 coordinate variables (WGS84 longitude, latitude and time stamp), as well as the following data variables:

• rzsm_1: (float) Root Zone Soil Moisture at 0-10 cm. Given in volumetric units [m3/m3].
• rzsm_2: (float) Root Zone Soil Moisture at 10-40 cm. Given in volumetric units [m3/m3].
• rzsm_3: (float) Root Zone Soil Moisture at 40-100 cm. Given in volumetric units [m3/m3].
• rzsm_4: (float) Root Zone Soil Moisture at 100-200. Given in volumetric units [m3/m3].
• uncertainty_1: (float) Root Zone Soil Moisture uncertainty at 0-10 cm from propagated surface uncertainties [m3/m3].
• uncertainty_2: (float) Root Zone Soil Moisture uncertainty at 10-40 cm from propagated surface uncertainties [m3/m3].
• uncertainty_3: (float) Root Zone Soil Moisture uncertainty at 40-100 cm from propagated surface uncertainties [m3/m3].
• uncertainty_4: (float) Root Zone Soil Moisture uncertainty at 100-200 cm from propagated surface uncertainties [m3/m3].

Additional information for each variable is given in the netCDF attributes.

Version Changelog

• v9.1
  • Initial version based on PASSIVE input data from ESA CCI SM v09.1 as used by Hirschi et al. (2025).

Software to open netCDF files

These data can be read by any software that supports Climate and Forecast (CF) conform metadata standards for netCDF files, such as:

• https://github.com/pydata/xarray" target="_blank" rel="noopener">Xarray (python)
• https://unidata.github.io/netcdf4-python/" target="_blank" rel="noopener">netCDF4 (python)
• https://github.com/TUW-GEO/esa_cci_sm">esa_cci_sm (python)
• Similar tools exists for other programming languages (Matlab, R, etc.)
• Software packages and GIS tools can open netCDF files, e.g. CDO, NCO, QGIS, ArCGIS
• You can also use the GUI software Panoply to view the contents of each file

References

• Dorigo, W., Wagner, W., Albergel, C., Albrecht, F., Balsamo, G., Brocca, L., Chung, D., Ertl, M., Forkel, M., Gruber, A., Haas, E., Hamer, P. D., Hirschi, M., Ikonen, J., de Jeu, R., Kidd, R., Lahoz, W., Liu, Y. Y., Miralles, D., Mistelbauer, T., Nicolai-Shaw, N., Parinussa, R., Pratola, C., Reimer, C., van der Schalie, R., Seneviratne, S. I., Smolander, T., and Lecomte, P.: ESA CCI Soil Moisture for improved Earth system understanding: State-of-the art and future directions, Remote Sensing of Environment, 203, 185-215, 10.1016/j.rse.2017.07.001, 2017
• Dorigo, W., Stradiotti, P., Preimesberger, W., Kidd, R., van der Schalie, R., Frederikse, T., Rodriguez-Fernandez, N., & Baghdadi, N. (2024). ESA Climate Change Initiative Plus - Soil Moisture Algorithm Theoretical Baseline Document (ATBD) Supporting Product Version 09.0. Zenodo. https://doi.org/10.5281/zenodo.13860922
• Gruber, A., Scanlon, T., van der Schalie, R., Wagner, W., and Dorigo, W.: Evolution of the ESA CCI Soil Moisture climate data records and their underlying merging methodology, Earth Syst. Sci. Data, 11, 717–739, https://doi.org/10.5194/essd-11-717-2019, 2019.
• Hirschi, M., Michel, D., Schumacher, D. L., Preimesberger, W., Seneviratne, S. I.: Recent summer soil moisture drying in Switzerland based on the SwissSMEX network, 2025 (paper submitted)
• Pasik, A., Gruber, A., Preimesberger, W., De Santis, D., and Dorigo, W.: Uncertainty estimation for a new exponential-filter-based long-term root-zone soil moisture dataset from Copernicus Climate Change Service (C3S) surface observations, Geosci. Model Dev., 16, 4957–4976, https://doi.org/10.5194/gmd-16-4957-2023, 2023
• Preimesberger, W., Stradiotti, P., and Dorigo, W.: ESA CCI Soil Moisture GAPFILLED: An independent global gap-free satellite climate data record with uncertainty estimates, Earth Syst. Sci. Data Discuss. [preprint], https://doi.org/10.5194/essd-2024-610, in review, 2025.

Related Records

Please see the ESA CCI Soil Moisture science data records community for more records based on ESA CCI SM.

This resource provides code examples for working directly with HydroShare S3 buckets (also referred to as HydroShare cloud storage buckets or simply HydroShare buckets) to access and manage resource files, without the need to download them locally first. Working directly with S3 buckets can offer better performance.

This resource includes the following notebooks: 1- hydroshare_s3_bucket_access_examples.ipynb: Examples for working directly with HydroShare S3 buckets to access and manage resource files. 2- hs_bucket_access_gdal_example.ipynb: Examples for reading raster and shapefile directly from HydroShare S3 buckets using gdal. 3- hs_bucket_access_non_gdal_example.ipynb: Examples of using h5netcdf and xarray for reading netcdf files, rioxarray for reading raster files, and pandas for reading CSV files, all directly from HydroShare S3 buckets.

This dataset is used as a "ground truth" for investigating the performance of a volumetric reconstruction technique of electric current densities, intended to be applied to the EISCAT 3D radar system. The technique is outlined in a mnuscript in preparation, to be referred to here once submitted. The volumetric reconstruction code can be found here: https://github.com/jpreistad/e3dsecs

This dataset contain four files:

1) Dataset file 'gemini_dataset.nc'. This is a dump from the end of a GEMINI model run driven with a pair of up/down FAC above the region around the EISCAT 3D facility. Detailes of the GEMINI model can be found here: https://doi.org/10.5281/zenodo.3528915 . This is a NETCDF file, intended to be opened with xarray in python:

import xaray

dataset = xarray.open_dataset('gemini_dataset.nc')

2) Grid file 'gemini_grid.h5'. This file is needed to get information about the grid that the values from GEMINI are represented in. The E3DSECS library (https://github.com/jpreistad/e3dsecs) has the necessary code to open this file and put it into the dictionary structure used in that package.

3) The GEMINI simulation config file 'config.nml' used to produce the simulation.

4) The GEMINI boundary file 'fac_said.py' used to produce the boundary conditions for the simulation

Together files 3 and 4 could be used to reproduce the full simulation of the GEMINI model, which is freely available at https://github.com/gemini3d

The configuration files for this particular run are also available at this location:

https://github.com/gemini3d/gemini-examples/tree/main/init/aurora_curv

Dataset used in the Galaxy Pangeo tutorials on Xarray.

Data is in netCDF format and is from Copernicus Air Monitoring Service and more precisely PM2.5 (Particle Matter < 2.5 μm) 4 days forecast from December, 22 2021. This dataset is very small and there is no need to parallelize our data analysis. Parallel data analysis with Pangeo is not covered in this tutorial and will make use of another dataset.

This dataset is not meant to be useful for scientific studies.

Background The Transonic Cascade TEAMAero (TCTA) is a compressor cascade test case designed at the German Aerospace Center (DLR) via numerical optimization for the TEAMAero research consortium. The cascade has a pitch of 65 mm and a stagger angle of 45.8° with respect to the horizontal axis. The profile shape is provided in this dataset as a .csv file. The cascade's aerodynamic design point has been computed by applying the DLR's TRACE discontinuous-Galerkin large eddy simulation solver for extended analysis. A subset of this data is provided in this repository for interested users. Data description One file containing the primitive variables sampled in the mid-span plane of the simulation domain. The data consists of a uniform grid of 489 by 1000 points (j, i) separated by 0.133 mm and covering one whole pitch (65 mm) of the flow field through the cascade extending from the inlet measurement plane (MP1) to the outlet measurement plane (MP2), see cited literature for more info. Approximately 3.86 ms or 14.4 convective time units (CTUs) of data are provided, sampled at 100 kHz over a total of 387 timesteps (t), starting from t=27 ms and corresponding to the last timesteps calculated in the simulation. The datasets were stored as HDF5 NetCDF files with the xarray Python package.. One csv file containing the profile of the TCTA blade tested in the TGK wind tunnel. Related material A Python tutorial with more details on the test case and the data is provided in the GitLab repository below: https://codebase.helmholtz.cloud/Edwin.MunozLopez/tcta-num-data-rep

netcdf files of hourly tilted radiation for 2016 (tilted_irrad_*.nc), based on MeteoSwiss solar radiation data (gridded) for Switzerland, as well as a file linking the rooftop IDs to rooftop properties (rooftop_link_info_v1.csv) and one to link them to communes in Switzerland (Sonnendach_Gemeinden_Dachflaechen.csv):

The tilted radiation files contain the following variables:

• tilted_irradiance with the hourly global tilted radiation (in W/m²), including direct, diffuse and reflected components as well as shading effects. Note that the radiation on flat roofs is computed for south-facing panels tilted at 30°.
• yearly_kWh_m2 with the annual global tilted radiation (in kWh/m²), i.e. the sum over time for each roof
• time with the actual hourly time stamp, which allows to recreate the "normal" time series from the timestamp/day format used in the files

and the following coordinates:

• DF_UID: the rooftop identifier (see roof info file explained below)
• timestamp: the "monthly-mean-hourly" time stamp, i.e. one hour for each month - this timestamp is used for compatibility with earlier versions of the dataset
• day: the day number (1-31) of each month - as written above, use the time variable to obtain the correct time stamp

The rooftop linking file (rooftop_link_info_v1.csv) contains the following relevant columns (plus a few others that are probably not of concern):

• DF_UID: the rooftop identifier (as above)
• SB_UUID: the SwissBuildings3D identifier, with which data can be grouped to buildings
• GWR_EGID_original: GWR_EGID as linked to SwissBuildings3D by Sonnendach.ch (swisstopo/meteotest)
• ROOF_AREA/TILT/ASPECT: Tilted roof area, roof tilt angle (°) and roof aspect angle (clockwise ° from south), as provided in Sonnendach.ch dataset (first version)
• XCOORD/YCOORD: Coordinates of geometry centroid (in Swiss LV03 coordinate system)
• The rest are information linked to the national buildings and dwellings registry (GWR) - I can provide more details if needed

The commune linking file (Sonnendach_Gemeinden_Dachflaechen.csv) contains the following columns:

• DF_UID, SB_UUID: see above
• BFS_NUMMER, NAME: number and name of the commune where the roof is located (see https://www.swisstopo.admin.ch/de/geodata/landscape/boundaries3d.html for more info)
• x_coord_WGS84,y_coord_WGS84: roof coordinates in WGS84 coordinate system

Some tips to work with the netcdf data:

• Open it using data = xr.open_mfdataset('tilted_irrad_*.nc') (requires the xarray and dask package, but avoids loading everything into memory and enables parallel processing of the data)
• The time variable is virtually copied since it is found in each batch. Replace it like data['time'] = data.time.isel(DF_UID = 0).drop('DF_UID')
• To use aggregated slices of the data, obtain a specific slice (i.e. roofs of a certain tilt and orientation, or in a certain commune) by calling data_slice = data.sel(DF_UID = [DF_UIDs of slice]).sum(dim = 'DF_UID') and convert it to a pandas dataframe like slice_df = data_slice.to_dataframe().set_index('time') - grouping in xarray does not work well, I would avoid it.
• To multiply them with another variable (e.g. IAM), this can be done very efficiently using xarray. If the respective variable has the same coordinates as the tilted radiation, the computation is done automatically in parallel and with low memory use when the file is saved, if xr.open_mfdataset() is used to load the data (everything is executed in batches using dask).