The Analysis Of Record for Calibration (AORC) is a gridded record of near-surface weather conditions covering the continental United States and Alaska and their hydrologically contributing areas. It is defined on a latitude/longitude spatial grid with a mesh length of 30 arc seconds (~800 m), and a temporal resolution of one hour. Elements include hourly total precipitation, temperature, specific humidity, terrain-level pressure, downward longwave and shortwave radiation, and west-east and south-north wind components. It spans the period from 1979 across the Continental U.S. (CONUS) and from 1981 across Alaska, to the near-present (at all locations). This suite of eight variables is sufficient to drive most land-surface and hydrologic models and is used as input to the National Water Model (NWM) retrospective simulation. While the native AORC process generates netCDF output, the data is post-processed to create a cloud optimized Zarr formatted equivalent for dissemination using cloud technology and infrastructure.

AORC Version 1.1 dataset creation
The AORC dataset was created after reviewing, identifying, and processing multiple large-scale, observation, and analysis datasets. There are two versions of The Analysis Of Record for Calibration (AORC) data.

The initial AORC Version 1.0 dataset was completed in November 2019 and consisted of a grid with 8 elements at a resolution of 30 arc seconds. The AORC version 1.1 dataset was created to address issues "see Table 1 in Fall et al., 2023" in the version 1.0 CONUS dataset. Full documentation on version 1.1 of the AORC data and the related journal publication are provided below.

The native AORC version 1.1 process creates a dataset that consists of netCDF files with the following dimensions: 1 hour, 4201 latitude values (ranging from 25.0 to 53.0), and 8401 longitude values (ranging from -125.0 to -67).

The data creation runs with a 10-day lag to ensure the inclusion of any corrections to the input Stage IV and NLDAS data.

Note - The full extent of the AORC grid as defined in its data files exceed those cited above; those outermost rows and columns of data grids are filled with missing values and are the remnant of an early set of required AORC extents that have since been adjusted inward.

AORC Version 1.1 Zarr Conversion

The goal for converting the AORC data from netCDF to Zarr was to allow users to quickly and efficiently load/use the data. For example, one year of data takes 28 mins to load via NetCDF while only taking 3.2 seconds to load via Zarr (resulting in a substantial increase in speed). For longer periods of time, the percentage increase in speed using Zarr (vs NetCDF) is even higher. Using Zarr also leads to less memory and CPU utilization.

It was determined that the optimal conversion for the data was 1 year worth of Zarr files with a chunk size of 18MB. The chunking was completed across all 8 variables. The chunks consist of the following dimensions: 144 time, 128 latitude, and 256 longitude. To create the files in the Zarr format, the NetCDF files were rechunked using chunk() and "Xarray". After chunking the files, they were converted to a monthly Zarr file. Then, each monthly Zarr file was combined using "to_zarr" to create a Zarr file that represents a full year

Users wanting more than 1 year of data will be able to utilize Zarr utilities/libraries to combine multiple years up to the span of the full data set.

There are eight variables representing the meteorological conditions
Total Precipitaion (APCP_surface)

1. Hourly total precipitation (kgm-2 or mm) for Calibration (AORC) dataset
   

Air Temperature (TMP_2maboveground)

1. Temperature (at 2 m above-ground-level (AGL)) (K)
   

Specific Humidity (SPFH_2maboveground)

1. Specific humidity (at 2 m AGL) (g g-1)
   

Downward Long-Wave Radiation Flux (DLWRF_surface)

1. longwave (infrared)
2. radiation flux (at the surface) (W m-2)
   

Downward Short-Wave Radiation Flux (DSWRF_surface)

1. Downward shortwave (solar)
2. radiation flux (at the surface) (W m-2)
   

Pressure (PRES_surface)

1. Air pressure (at the surface) (Pa)
   

**U-Component of Wind (UGRD_10maboveground)"
1)U (west-east) - components of the wind (at 10 m AGL) (m s-1)
**V-Component of Wind (VGRD_10maboveground)"

1. V (south-north) - components of the wind (at 10 m AGL) (m s-1)
   
   

Precipitation and Temperature

The gridded AORC precipitation dataset contains one-hour Accumulated Surface Precipitation (APCP) ending at the “top” of each hour, in liquid water-equivalent units (kg m-2 to the nearest 0.1 kg m-2), while the gridded AORC temperature dataset is comprised of instantaneous, 2 m above-ground-level (AGL) temperatures at the top of each hour (in Kelvin, to the nearest 0.1).

Specific Humidity, Pressure, Downward Radiation, Wind

The development process for the six additional dataset components of the Conus AORC [i.e., specific humidity at 2m above ground (kg kg-1); downward longwave and shortwave radiation fluxes at the surface (W m-2); terrain-level pressure (Pa); and west-east and south-north wind components at 10 m above ground (m s-1)] has two distinct periods, based on datasets and methodology applied: 1979–2015 and 2016–present.

NOAA National Weather Service (NWS) Historical Hydrology, Analysis of Record for Calibration (AORC), air temperature data for the AORC ABRFC (Oklahoma) region, using data from https://hydrology.nws.noaa.gov/aorc-historic/AORC_ABRFC_4km/ . Instantaneous 2-m above ground temperature valid at nominal file time, and 1-h precipitation accumulation ending at the nominal file time. The period covered is 5 January 1979 to 31 December 2017. Data are stored at a precision of 0.1K and 0.1mm. Data are unprojected, that is, defined by a latitude/longitude grid. File collections on the ftp server are from a thinned AORC grid at a spacing of 0.032 degrees, approximately 4-km mesh.

These data are atmospheric forcings from NOAA's Analysis of Record for Calibration (AORC 2024) summarized over the CAMELS catchments (Addor et al. 2017), however, the delineations over those CAMELS basins were updated by Andy Wood (2024) using the basin boundaries produced for the USGS GagesII effort (Falcone 2011).

Data collected using this repository code: https://github.com/jmframe/CIROH_DL_NextGen/tree/main/forcing_prep, which should be somewhat derivative of this: https://github.com/CIROH-UA/ngen-datastream/tree/main/forcingprocessor.

There are eight variables representing the meteorological conditions: Total Precipitaion (APCP_surface) - Hourly total precipitation (kgm-2 or mm) for Calibration (AORC) dataset Air Temperature (TMP_2maboveground) - Temperature (at 2 m above-ground-level (AGL)) (K) Specific Humidity (SPFH_2maboveground) - Specific humidity (at 2 m AGL) (g g-1) Downward Long-Wave Radiation Flux (DLWRF_surface) - longwave (infrared) radiation flux (at the surface) (W m-2) Downward Short-Wave Radiation Flux (DSWRF_surface) - Downward shortwave (solar) radiation flux (at the surface) (W m-2) Pressure (PRES_surface) - Air pressure (at the surface) (Pa) U-Component of Wind (UGRD_10maboveground) - (west-east) - components of the wind (at 10 m AGL) (m s-1) V-Component of Wind (VGRD_10maboveground) - (south-north) - components of the wind (at 10 m AGL) (m s-1)

The data include a timestamp from Coordinated Universal Time (UTC; AORC 2021).

References: Addor, N., Newman, A. J., Mizukami, N., and Clark, M. P.: The CAMELS data set: catchment attributes and meteorology for large-sample studies, Hydrol. Earth Syst. Sci., 21, 5293–5313, https://doi.org/10.5194/hess-21-5293-2017, 2017. AORC. 2021. Analysis of Record for Calibration Version 1.1 - Sources, Methods, and Verification, National Oceanic and Atmospheric Administration (NOAA), National Weather Service (NWS), Office of Water Prediction (OWP), Silver Spring, MD.https://www.weather.gov/media/owp/operations/aorc_v1_1_methods.pd AORC. 2024. NOAA Analysis of Record for Calibration (AORC) Dataset was accessed July 2024 from https://registry.opendata.aws/noaa-nws-aorc. Falcone, J., 2011, GAGES-II: Geospatial Attributes of Gages for Evaluating Streamflow: U.S. Geological Survey data release, https://doi.org/10.5066/P96CPHOT. https://www.sciencebase.gov/catalog/item/631405bbd34e36012efa304a Wood. 2024. CAMELS basins delineation.

The Analysis of Record Calibration (AORC) meteorological forcing dataset is a retrospective gridded product produced by NOAA’s Office of Water Prediction (Fall et al., 2023). The AORC hourly dataset provides high resolution (800-m) gridded fields for seven meteorological variables (‘forcings’), used as time series inputs for calibration of the current NOAA National Water Model (NWMv3), among other modeling and analysis applications. The upcoming v4 update of the NWM will adopt a catchment-based modeling approach centered on an NHDPlus-based NextGen Hydrofabric (Johnson et al., 2024).

In view of this transition, we provide a remapped daily version of the AORC forcings (with the addition of maximum and minimum temperature) to the Hydrofabric catchments representing a large-sample watershed subset – i.e., for the 671 CAMELS basins (Newman et al., 2015; Addor et al., 2017; see https://ral.ucar.edu/solutions/products/camels). The associated CAMELS-Hydrofabric v2.2 data are provided as a separate Hydroshare resource, "Nextgen-CAMELS-Hydrofabric v2.2: NOAA Next Generation Water Resources Modeling Framework Hydrofabric for the CAMELS Basins". The remapping was performed by calculating catchment areal-mean forcing values using a spatially-conservative remapping technique developed previously at the NSF National Center for Atmospheric Research (NCAR). Specifically, the ‘poly2poly.py’ script is used to generate spatial weights relating the AORC grid to the catchment Hydrofabric polygons, and associated scripts then apply the weights efficiently over all the timesteps in the dataset. Catchment averages are adjusted to account for missing input grid values, where necessary. The remapped dataset of n = 54,667 catchments is available in netCDF format and designed to meets the standards of the Next Generation Water Resources Modeling Framework (Nextgen) – such as variable naming and epoch start date formatting – making it model-ready off the shelf. This retrospective 1979-2023 forcing dataset, together with the hydrofabric divides, nexus, and flowpath layers, provides a core input required for running the Nextgen modeling framework at a Hydrofabric resolution across the CAMELS basins. Note, an earlier lumped-basin version of these forcings is available at: https://www.hydroshare.org/resource/c738c05278a34bc9848dd14d61cffab9/ (Frame et al, 2024).

Acknowledgements: - This dataset was supported by a project grant to the Colorado School of Mines (CSM) from the NOAA Cooperative Institute for Research to Operations in Hydrology (CIROH). CIROH is funded via the NOAA Cooperative Agreement with The University of Alabama (NA22NWS4320003). - We are grateful to Mike Robbert at CSM for his assistance in transferring the large AORC dataset onto Mines HPC systems to facilitate the remapping process.

References: Fall, Greg, David Kitzmiller, Sandra Pavlovic, Ziya Zhang, Nathan Patrick, Michael St. Laurent, Carl Trypaluk, Wanru Wu, and Dennis Miller. “The Office of Water Prediction’s Analysis of Record for Calibration, Version 1.1: Dataset Description and Precipitation Evaluation,” December 2023. https://doi.org/10.1111/1752-1688.13143. Addor, N., Newman, A. J., Mizukami, N. and Clark, M. P.: The CAMELS data set: catchment attributes and meteorology for large-sample studies, Hydrol. Earth Syst. Sci., 21, 5293–5313, doi:10.5194/hess-21-5293-2017, 2017. Frame, J. M., A. W. Wood, N. Frazier (2024). AORC atmospheric forcing data across CAMELS US basins, 1980 - 2024, HydroShare, http://www.hydroshare.org/resource/c738c05278a34bc9848dd14d61cffab9 Johnson, J. Michael, Arash Modesari Rad, Trey C. Flowers, and Fred L. Ogden. “The NOAA Next Generation Water Resource Modeling Framework Hydrofabric.” In 104th AMS Annual Meeting. AMS, 2024. https://ams.confex.com/ams/104ANNUAL/meetingapp.cgi/Paper/436827. Newman, A. J., Clark, M. P., Sampson, K., Wood, A., Hay, L. E., Bock, A., Viger, R. J., Blodgett, D., Brekke, L., Arnold, J. R., Hopson, T., and Duan, Q., 2015: Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance, Hydrol. Earth Syst. Sci., 19, 209–223, https://doi.org/10.5194/hess-19-209-2015 . Newman, A. J., N. Mizukami, M. P. Clark, A. W. Wood, B. Nijssen, and G. Nearing, 2017: Benchmarking of a Physically Based Hydrologic Model. J. Hydrometeor., 18, 2215–2225, https://doi.org/10.1175/JHM-D-16-0284.1. NOAA Analysis of Record for Calibration (AORC) Dataset was accessed on 2024-11-07 from https://registry.opendata.aws/noaa-nws-aorc.

The NOAA National Water Model Retrospective dataset contains input and output from multi-decade CONUS retrospective simulations. These simulations used meteorological input fields from meteorological retrospective datasets. The output frequency and fields available in this historical NWM dataset differ from those contained in the real-time operational NWM forecast model. Additionally, note that no streamflow or other data assimilation is performed within any of the NWM retrospective simulations

One application of this dataset is to provide historical context to current near real-time streamflow, soil moisture and snowpack conditions. The retrospective data can be used to infer flow frequencies and perform temporal analyses with hourly streamflow output and 3-hourly land surface output. This dataset can also be used in the development of end user applications which require a long baseline of data for system training or verification purposes.


Details for Each Version of the NWM Retrospective Output

CONUS Domain - CONUS retrospective output is provided by all four versions of the NWM

1. Version 3.0 - A 44-year (February 1979 through January 2023) retrospective simulation using version 3.0 of the National Water Model.
2. Version 2.1 - A 42-year (February 1979 through December 2020) retrospective simulation using version 2.1 of the National Water Model.
3. Version 2.0 - A 26-year (January 1993 through December 2018) retrospective simulation using version 2.0 of the National Water Model.
4. Version 1.2 - A 25-year (January 1993 through December 2017) retrospective simulation using version 1.2 of the National Water Model.
   
   

oCOUNS Domains - Only v3.0 of the NWM Retrospective data set provides coverage of the NWM Alaska, Hawaii and Puerto Rico / US Virgin Island domains.

CONUS Meteorological Forcing

1. Versions 3.0 and 2.1: NWM Retrospective simulations used forcing from the Office of Water Prediction Analysis of Record for Calibration (AORC) dataset. NWM v2.1 used AORC v1.0 for 1979-2006 and AORC v1.1 for 2007-2020, while NWM v3.0 used AORC v1.1 for the full v3.0 (1979-2023 period)

Important Warning - While the metadata tag in the NWM v3.0 forcing files label the files as “v2.1”, the files are in fact v3.0 forcing files. 2) Versions 2.0 and 1.2 - NWM Retrospective simulation uses forcing from the North American Land Data Assimilation System (NLDAS) dataset
oCONUS Meteorological Forcing

1. Version 3.0 - AORC Alaska forcing was used to drive the NWM Alaska simulation, while North American Regional Reanalysis (NARR) data along with precipitation from the Alaska Pacific River Forecast Center (APRFC) was used to drive the Hawaii retrospective simulation. Similarly, the Puerto Rico / US Virgin Island retrospective simulation was driven by NARR data along with precipitation from the Southeast River Forecast Center.
   
   

Formats - NWM Retrospective data is available in two formats, NetCDF and Zarr. The NetCDF files contain the full set of NWM output data, while the Zarr files contain a subset of NWM output fields that vary with model version.

1. NWM V3.0: All model output and forcing input fields are available in the NetCDF format. All model output fields along with the precipitation forcing field are available in the Zarr format.
2. NWM V2.1: All model output and forcing input fields are available in the NetCDF format. Many of the model output fields along with the precipitation forcing field are available in the Zarr format
3. NWM V2.0: All model output fields are available in NetCDF format. Model channel output including streamflow and related fields are available in Zarr format.
4. NWM V1.2: All model output fields are available in NetCDF format.
   
   A table listing the data available within each NetCDF and Zarr file is located in the 'documentation page'. This data includes meteorological NWM forcing inputs along with NWM hydrologic and land surface outputs, and varies by version number.

This HydroShare resource contains Jupyter Notebooks with instructions and code for accessing and subsetting the NOAA Analysis of Record for Calibration (AORC) dataset. The resource includes two Jupyter Notebooks: 1. AORC_Point_Data_Retrieval.ipynb: Retrieves data for a specific point within the U.S. AORC coverage area, specified using geographic coordinates. 2. AORC_Zone_Data_Retrieval.ipynb: Retrieves data for an area defined by an uploaded polygon shapefile. These notebooks programmatically retrieve the data from Amazon Web Services (https://registry.opendata.aws/noaa-nws-aorc/) , aggregate data at user-defined time scales (which may differ from NOAA’s original time steps), and, in the case of shapefile-based data retrieval, compute the average over the shapes in the given shapefile. The provided notebooks are coded to retrieve data from AORC version 1.1 released in ZARR format in December 2023.

The Analysis Of Record for Calibration (AORC) is a gridded record of near-surface weather conditions covering the continental United States and Alaska and their hydrologically contributing areas (https://registry.opendata.aws/noaa-nws-aorc/). It is defined on a latitude/longitude spatial grid with a mesh length of 30 arc seconds (~800 m), and a temporal resolution of one hour. Elements include hourly total precipitation, temperature, specific humidity, terrain-level pressure, downward longwave and shortwave radiation, and west-east and south-north wind components. It spans the period from 1979 across the Continental U.S. (CONUS) and from 1981 across Alaska, to the near-present (at all locations). This suite of eight variables is sufficient to drive most land-surface and hydrologic models and is used as input to the National Water Model (NWM) retrospective simulation. While the original NOAA process generated AORC data in netCDF format, the data has been post-processed to create a cloud optimized Zarr formatted equivalent that NOAA also disseminates.

The objective of this HydroShare resource is to query AORC v1.0 Forcing data stored on HydroShare's Thredds server and create a subset of this dataset for a designated watershed and timeframe. The user is prompted to define their temporal and spatial frames of interest, which specifies the start and end dates for the data subset. Additionally, the user is prompted to define a spatial frame of interest, which could be a bounding box or a shapefile, to subset the data spatially.

Before the subsetting is performed, data is queried, and geospatial metadata is added to ensure that the data is correctly aligned with its corresponding location on the Earth's surface. To achieve this, two separate notebooks were created - this notebook and this notebook - which explain how to query the dataset and add geospatial metadata to AORC v1.0 data in detail, respectively. In this notebook, we call functions from the AORC.py script to perform these preprocessing steps, resulting in a cleaner notebook that focuses solely on the subsetting process.

This resource contains a small subset of AORC v1.1 data that is being used to test programatic access (and subsetting) via Python and Thredds. The AORC data was uploaded as a zip archive and extracted inside HydroShare. When this operation initiated, HydroShare automatically created "multidimensional" content aggregations for each NetCDF file within the archive. Note that this only works if the files contain a *.nc extension, which was added prior to uploading to HydroShare. The aorc-hs-thredds.ipynb notebook demonstrates how these data can be queried directly from the HydroShare Thredds server.

This study assesses snow water equivalent (SWE) simulation uncertainty in the National Water Model (NWM) due to forcing and model parameterization, using data from 46 Snow Telemetry (SNOTEL) sites in the Upper Colorado River Basin (UCRB). We evaluated the newly developed Analysis of Record for Calibration (AORC) forcing data for SWE simulation and examined the impact of bias correction applied to AORC precipitation and temperature. Additionally, we investigated the sensitivity of SWE simulations to choices of physical parameterization schemes through 72 ensemble experiments. Results showed that NWM driven by AORC forcings captured the overall temporal variation of SWE but underestimated its amount. Adjusting AORC precipitation with SNOTEL observations reduced SWE root-mean-square error (RMSE) by 66%, adjusting temperature trimmed it by 10%, and adjusting both decreased it by 69%. Among the physical processes, the snow/soil temperature time scheme (STC) demonstrated the highest sensitivity, followed by the surface exchange coefficient for heat (SFC), snow surface albedo (ALB), and rainfall and snowfall partitioning (SNF), while the lower boundary of soil temperature (TBOT) proved to be insensitive. Further optimization of the parameterization combination resulted in a 12% SWE RMSE reduction. When combined with the bias-corrected AORC precipitation and temperature, this optimization led to a remarkable 78% SWE RMSE reduction. Despite these enhancements, a persistent slow and late spring ablation suggests model deficiencies in snow ablation physics. The study emphasizes the critical need to enhance the accuracy of forcing data in mountainous regions and address model parameterization uncertainty through optimization efforts.

AORC Forcing Data from AWS subsetted to the Great Salt Lake basin. This is data for one day: 01/01/2016

Since the release of the CAMELS large-sample catchment dataset a decade ago, the CAMELS watershed collection has become a de facto set of minimally impacted headwater basins for hydrologic model benchmarking studies in the continental US (Newman et al., 2015; 2017; see https://ral.ucar.edu/solutions/products/camels). In this dataset, we release the first subset of the Next Generation Water Resources Modeling Framework (Nextgen) Hydrofabric (v2.2; Johnson, 2022; Johnson et al., 2024) that includes the full spatial extent of the CAMELS basin delineations, after revision to correct erroneous boundaries (by using more reliable published Gages-II shapefiles). Our Hydrofabric subsetting workflow includes all Nextgen catchments that are within (with at least 5% area overlap) and/or hydrologically connected to the updated CAMELS basins shapes, thus this catchment boundary dataset enables a wide range of possible use cases, e.g., streamflow routing with alternative river networks. It also ensures improved comparison of hydrologic simulations to other CAMELS model benchmarking studies that do not use the Nextgen Hydrofabric. The layers included in this dataset provide the key geospatial inputs for Nextgen, while the accompanying daily 1979-2023 AORC retrospective forcing dataset (https://www.hydroshare.org/resource/1e934d92acdb42fa8379cd5009371150/; Sturtevant and Wood, 2024) also contributes to the required set of inputs needed for running the Nextgen modeling framework on the Hydrofabric v2.2 spatial units across the 671 CAMELS basins.

Acknowledgements: This dataset was supported by a project grant to the Colorado School of Mines from the NOAA Cooperative Institute for Research to Operations in Hydrology (CIROH). CIROH is funded via the NOAA Cooperative Agreement with The University of Alabama (NA22NWS4320003). The revised CAMELS boundary GIS work was supported by the US Army Corps of Engineers Climate Preparedness and Resilience Program via a grant to NSF NCAR. We also acknowledge and thank Mike Johnson and Lynker Spatial for use of and assistance with the Hydrofabric v2.2 dataset

References: Newman, A. J., Clark, M. P., Sampson, K., Wood, A., Hay, L. E., Bock, A., Viger, R. J., Blodgett, D., Brekke, L., Arnold, J. R., Hopson, T., and Duan, Q. (2015). Development of a large-sample watershed-scale hydrometeorological data set for the contiguous USA: data set characteristics and assessment of regional variability in hydrologic model performance, Hydrol. Earth Syst. Sci., 19, 209–223, https://doi.org/10.5194/hess-19-209-2015 . Newman, A. J., N. Mizukami, M. P. Clark, A. W. Wood, B. Nijssen, and G. Nearing (2017). Benchmarking of a Physically Based Hydrologic Model. J. Hydrometeor., 18, 2215–2225, https://doi.org/10.1175/JHM-D-16-0284.1. Johnson, J. M. (2022). National Hydrologic Geospatial Fabric (hydrofabric) for the Next Generation (NextGen) Hydrologic Modeling Framework, HydroShare http://www.hydroshare.org/resource/129787b468aa4d55ace7b124ed27dbde Johnson, J. Michael, Arash Modesari Rad, Trey C. Flowers, and Fred L. Ogden. (2024). “The NOAA Next Generation Water Resource Modeling Framework Hydrofabric.” In 104th AMS Annual Meeting. AMS, 2024. https://ams.confex.com/ams/104ANNUAL/meetingapp.cgi/Paper/436827. Sturtevant, J. and Wood, A. (2024). Daily AORC meteorological forcings (1979-2023) for Nextgen-CAMELS-Hydrofabric Catchments (v2.2), HydroShare, http://www.hydroshare.org/resource/1e934d92acdb42fa8379cd5009371150

This resource was first created for a live demo during an online I-GUIDE VCO meeting on May 9, 2023. It was then modified for another live demo during the 1st annual CIROH users and developers conference in Salt Lake City, May 16-18. Recently, it was used for the National Water Center Summer Institute 2023.

It contains codes and inputs for a precipitation analysis across the Logan River Watershed. In this analysis, we will obtain modeled precipitation from two products: AORC and PRISM, compare the basin's average daily precipitation, and save results back to HydroShare.

This resource provides codes and inputs for comparing basin-averaged AORC precipitation datasets over the Logan River Watershed.