SCALABLE TIME SERIES CHANGE DETECTION FOR BIOMASS MONITORING USING GAUSSIAN PROCESS VARUN CHANDOLA AND RANGA RAJU VATSAVAI Abstract. Biomass monitoring, specifically, detecting changes in the biomass or vegetation of a geographical region, is vital for studying the carbon cycle of the system and has significant implications in the context of understanding climate change and its impacts. Recently, several time series change detection methods have been proposed to identify land cover changes in temporal profiles (time series) of vegetation collected using remote sensing instruments. In this paper, we adapt Gaussian process regression to detect changes in such time series in an online fashion. While Gaussian process (GP) has been widely used as a kernel based learning method for regression and classification, their applicability to massive spatio-temporal data sets, such as remote sensing data, has been limited owing to the high computational costs involved. In our previous work we proposed an efficient Toeplitz matrix based solution for scalable GP parameter estimation. In this paper we apply these solutions to a GP based change detection algorithm. The proposed change detection algorithm requires a memory footprint which is linear in the length of the input time series and runs in time which is quadratic to the length of the input time series. Experimental results show that both serial and parallel implementations of our proposed method achieve significant speedups over the serial implementation. Finally, we demonstrate the effectiveness of the proposed change detection method in identifying changes in Normalized Difference Vegetation Index (NDVI) data.

Time Series Analysis Software Market size was valued at USD 1.8 Billion in 2024 and is projected to reach USD 4.7 Billion by 2032, growing at a CAGR of 10.5% during the forecast period 2026-2032.

Global Time Series Analysis Software Market Drivers

Growing Data Volumes: The exponential growth in data generated across various industries necessitates advanced tools for analyzing time series data. Businesses need to extract actionable insights from large datasets to make informed decisions, driving the demand for time series analysis software.

Increasing Adoption of IoT and Connected Devices: The proliferation of Internet of Things (IoT) devices generates continuous streams of time-stamped data. Analyzing this data in real-time helps businesses optimize operations, predict maintenance needs, and enhance overall efficiency, fueling the demand for time series analysis tools.

Advancements in Machine Learning and AI: Integration of machine learning and artificial intelligence (AI) with time series analysis enhances predictive capabilities and automates the analysis process. These advancements enable more accurate forecasting and anomaly detection, attracting businesses to adopt sophisticated analysis software.

Need for Predictive Analytics: Businesses are increasingly focusing on predictive analytics to anticipate future trends and behaviors. Time series analysis is crucial for forecasting demand, financial performance, stock prices, and other metrics, driving the market growth.

Industry 4.0 and Automation: The push towards Industry 4.0 involves automating industrial processes and integrating smart technologies. Time series analysis software is essential for monitoring and optimizing manufacturing processes, predictive maintenance, and supply chain management in this context.

Financial Sector Growth: The financial industry extensively uses time series analysis for modeling stock prices, risk management, and economic forecasting. The growing complexity of financial markets and the need for real-time data analysis bolster the demand for specialized software.

Healthcare and Biomedical Applications: Time series analysis is increasingly used in healthcare for monitoring patient vitals, managing medical devices, and analyzing epidemiological data. The focus on personalized medicine and remote patient monitoring drives the adoption of these tools.

Climate and Environmental Monitoring: Governments and organizations use time series analysis to monitor climate change, weather patterns, and environmental data. The need for accurate predictions and real-time monitoring in environmental science boosts the market.

Regulatory Compliance and Risk Management: Industries such as finance, healthcare, and energy face stringent regulatory requirements. Time series analysis software helps in compliance by providing detailed monitoring and reporting capabilities, reducing risks associated with regulatory breaches.

Emergence of Big Data and Cloud Computing: The adoption of big data technologies and cloud computing facilitates the storage and analysis of large volumes of time series data. Cloud-based time series analysis software offers scalability, flexibility, and cost-efficiency, making it accessible to a broader range of businesses.

Multivariate Time-Series (MTS) are ubiquitous, and are generated in areas as disparate as sensor recordings in aerospace systems, music and video streams, medical monitoring, and financial systems. Domain experts are often interested in searching for interesting multivariate patterns from these MTS databases which can contain up to several gigabytes of data. Surprisingly, research on MTS search is very limited. Most existing work only supports queries with the same length of data, or queries on a fixed set of variables. In this paper, we propose an efficient and flexible subsequence search framework for massive MTS databases, that, for the first time, enables querying on any subset of variables with arbitrary time delays between them. We propose two provably correct algorithms to solve this problem — (1) an R-tree Based Search (RBS) which uses Minimum Bounding Rectangles (MBR) to organize the subsequences, and (2) a List Based Search (LBS) algorithm which uses sorted lists for indexing. We demonstrate the performance of these algorithms using two large MTS databases from the aviation domain, each containing several millions of observations. Both these tests show that our algorithms have very high prune rates (>95%) thus needing actual disk access for only less than 5% of the observations. To the best of our knowledge, this is the first flexible MTS search algorithm capable of subsequence search on any subset of variables. Moreover, MTS subsequence search has never been attempted on datasets of the size we have used in this paper.

This dataset comprises 524 recordings of 6-dimensional time series data, capturing forces in three directions and torques in three directions during the assembly of small car model wheels. The data was collected using an equidistant sampling method with a sampling period of 0.004 seconds. Each time series represents the process of assembling one wheel, specifically the placement of a tire onto a rim, and includes a label indicating whether the assembly was successful (OK). The wheels were assembled in batches of four, and the recordings were obtained over six different days. The labels of recordings from two (days 3 and 4) of the six days are invalid as described in [1]. The labels presented in this data set are only binary (they do not describe the reason of the failure). The labels of recordings from days 5 and 6 are created by human while the other labels came from a convolutional neural network based computer vision classifier and can be inaccurate as described in section 5.4 of [1].

Dataset Structure:

File: ForceTorqueTimeSeries.csv

Columns:

idx (1-524): Index of the recording corresponding to the assembly of one wheel.

label (true/false): Indicates whether the assembly was successful (TRUE = product is OK).

meas_id (1-6): Identifier for the day on which the recording was made (refer to Table 2.1 in [1]).

force_x: X-component of the force measured by the sensor mounted on the delta robot's end effector.

force_y: Y-component of the force.

force_z: Z-component of the force.

torque_x: X-component of the torque.

torque_y: Y-component of the torque.

torque_z: Z-component of the torque.

Additional Files:

IMG_3351.MOV: A video demonstrating the assembly process for one batch of four wheels.

F3-BP-2024-Trna-Ales-Ales Trna - 2024 - Anomaly detection in robotic assembly process using force and torque sensors.pdf: Bachelor thesis [1] detailing the dataset and preliminary experiments on fault detection.

F3-BP-2024-Hanzlik-Vojtech-Anomaly_Detection_Bachelors_Thesis.pdf: Bachelor thesis [2] describing the data acquisition process.

References:

Trna, A. (2024). Anomaly detection in robotic assembly process using force and torque sensors [Bachelor’s thesis, Czech Technical University in Prague].

Hanzlik, V. (2024). Edge AI integration for anomaly detection in assembly using Delta robot [Bachelor’s thesis, Czech Technical University in Prague].

This lesson was adapted from educational material written by Dr. Kateri Salk for her Fall 2019 Hydrologic Data Analysis course at Duke University. This is the first part of a two-part exercise focusing on time series analysis.

Introduction

Time series are a special class of dataset, where a response variable is tracked over time. The frequency of measurement and the timespan of the dataset can vary widely. At its most simple, a time series model includes an explanatory time component and a response variable. Mixed models can include additional explanatory variables (check out the nlme and lme4 R packages). We will be covering a few simple applications of time series analysis in these lessons.

Opportunities

Analysis of time series presents several opportunities. In aquatic sciences, some of the most common questions we can answer with time series modeling are:

• Has there been an increasing or decreasing trend in the response variable over time?

• Can we forecast conditions in the future?

  Challenges

Time series datasets come with several caveats, which need to be addressed in order to effectively model the system. A few common challenges that arise (and can occur together within a single dataset) are:

• Autocorrelation: Data points are not independent from one another (i.e., the measurement at a given time point is dependent on previous time point(s)).

• Data gaps: Data are not collected at regular intervals, necessitating interpolation between measurements. There are often gaps between monitoring periods. For many time series analyses, we need equally spaced points.

• Seasonality: Cyclic patterns in variables occur at regular intervals, impeding clear interpretation of a monotonic (unidirectional) trend. Ex. We can assume that summer temperatures are higher.

• Heteroscedasticity: The variance of the time series is not constant over time.

• Covariance: the covariance of the time series is not constant over time. Many of these models assume that the variance and covariance are similar over the time-->heteroschedasticity.

  Learning Objectives

After successfully completing this notebook, you will be able to:

1. Choose appropriate time series analyses for trend detection and forecasting

2. Discuss the influence of seasonality on time series analysis

3. Interpret and communicate results of time series analyses

Multivariate Time-Series (MTS) are ubiquitous, and are generated in areas as disparate as sensor recordings in aerospace systems, music and video streams, medical monitoring, and financial systems. Domain experts are often interested in searching for interesting multivariate patterns from these MTS databases which can contain up to several gigabytes of data. Surprisingly, research on MTS search is very limited. Most existing work only supports queries with the same length of data, or queries on a fixed set of variables. In this paper, we propose an efficient and flexible subsequence search framework for massive MTS databases, that, for the first time, enables querying on any subset of variables with arbitrary time delays between them. We propose two provably correct algorithms to solve this problem — (1) an R-tree Based Search (RBS) which uses Minimum Bounding Rectangles (MBR) to organize the subsequences, and (2) a List Based Search (LBS) algorithm which uses sorted lists for indexing. We demonstrate the performance of these algorithms using two large MTS databases from the aviation domain, each containing several millions of observations. Both these tests show that our algorithms have very high prune rates (>95%) thus needing actual disk access for only less than 5% of the observations. To the best of our knowledge, this is the first flexible MTS search algorithm capable of subsequence search on any subset of variables. Moreover, MTS subsequence search has never been attempted on datasets of the size we have used in this paper.

Time series of the October mean ozone vertical column above Antarctica.

The dataset behind those images was developed within the ESA Climate Change Initiative Programme (https://climate.esa.int/en/projects/ozone/) and is operationally distributed by the EU Copernicus Climate Change Service

Abstract Prognostics solutions for mission critical systems require a comprehensive methodology for proactively detecting and isolating failures, recommending and guiding condition-based maintenance actions, and estimating in real time the remaining useful life of critical components and associated subsystems. A major challenge has been to extend the benefits of prognostics to include computer servers and other electronic components. The key enabler for prognostics capabilities is monitoring time series signals relating to the health of executing components and subsystems. Time series signals are processed in real time using pattern recognition for proactive anomaly detection and for remaining useful life estimation. Examples will be presented of the use of pattern recognition techniques for early detection of a number of mechanisms that are known to cause failures in electronic systems, including: environmental issues; software aging; degraded or failed sensors; degradation of hardware components; degradation of mechanical, electronic, and optical interconnects. Prognostics pattern classification is helping to substantially increase component reliability margins and system availability goals while reducing costly sources of "no trouble found" events that have become a significant warranty-cost issue. Bios Aleksey Urmanov is a research scientist at Sun Microsystems. He earned his doctoral degree in Nuclear Engineering at the University of Tennessee in 2002. Dr. Urmanov's research activities are centered around his interest in pattern recognition, statistical learning theory and ill-posed problems in engineering. His most recent activities at Sun focus on developing health monitoring and prognostics methods for EP-enabled computer servers. He is a founder and an Editor of the Journal of Pattern Recognition Research. Anton Bougaev holds a M.S. and a Ph.D. degrees in Nuclear Engineering from Purdue University. Before joining Sun Microsystems Inc. in 2007, he was a lecturer in Nuclear Engineering Department and a member of Applied Intelligent Systems Laboratory (AISL), of Purdue University, West Lafayette, USA. Dr. Bougaev is a founder and the Editor-in-Chief of the Journal of Pattern Recognition Research. His current focus is in reliability physics with emphasis on complex system analysis and the physics of failures which are based on the data driven pattern recognition techniques.

Fish Monitoring Dataset from the Okavango Delta, collected by Okavango Research Institute. Data Licenses as per ORI and JRS Biodiversity data standards aggreement.

The size and share of this market is categorized based on Application (Relational Databases, NoSQL Databases, Specialized Time Series Databases) and Product (Time-Based Data Storage, Analytics, Monitoring Systems, IoT Applications) and geographical regions (North America, Europe, Asia-Pacific, South America, Middle-East and Africa).

Time Series Databases Software Market size was valued at USD 359.37 USD Million in 2024 and is projected to reach USD 773.71 Million by 2031, growing at a CAGR of 10.06% from 2024 to 2031.

Time Series Databases Software Market Drivers

Growing Data Volume: The exponential growth of data generated by various sources, including IoT devices, financial transactions, and digital services, necessitates efficient management and analysis of time-stamped data. Time series databases are optimized for handling large volumes of time-stamped data, driving their adoption.

Rise of IoT and Connected Devices: The proliferation of IoT devices in industries such as manufacturing, healthcare, and smart cities generates massive amounts of time-series data. Time series databases are crucial for storing, querying, and analyzing this continuous stream of data efficiently.

Increasing Importance of Real-Time Analytics: Businesses require real-time insights to make informed decisions and maintain competitive advantage. Time series databases support real-time analytics by efficiently processing and analyzing time-stamped data, which is critical for applications like monitoring, forecasting, and anomaly detection.

This dataset contains time series measurements of temperature and salinity at the GAK1 site at the mouth of Resurrection Bay near Seward, AK from December 1999 through October 2002. Instrument packages were deployed at 6 depth levels.

A validation assessment of Land Cover Monitoring, Assessment, and Projection Collection 1.1 annual land cover products (1985–2019) for the Conterminous United States was conducted with an independently collected reference dataset. Reference data land cover attributes were assigned by trained interpreters for each year of the time series (1984–2018) to a reference sample of 24,971 randomly-selected Landsat resolution (30m x 30m) pixels. The LCMAP and reference dataset labels for each pixel location are displayed here for each year, 1985–2018.

You are not authorized to view this dataset. You may email the responsible party OEAW to request access.

This repository contains a novel time-series dataset for impact detection and localization on a plastic thin-plate, towards Structural Health Monitoring applications, using ceramic piezoelectric transducers (PZTs) connected to an Internet of Things (IoT) device. The dataset was collected from an experimental procedure of low-velocity, low-energy impact events that includes at least 3 repetitions for each unique experiment, while the input measurements come from 4 PZT sensors placed at the corners of the plate. For each repetition and sensor, 5000 values are stored with 100 KHz sampling rate. The system is excited with a steel ball, and the height from which it is released varies from 10 cm to 20 cm.

To the best of our knowledge, we are the first, to publish a public dataset that contains PZT sensors measurements concerning low-velocity, low-energy impact events in a thin plastic plate. In addition, we also contribute with our methodology on data collection using an SHM IoT system with resource constraints (based on Arduino NANO 33 MCU), as opposed to the majority of the literature that uses Oscilloscopes for data acquisition. This concept of an MCU-based system for data collection in SHM is especially important nowadays, due to the fast rise of extreme-edge and embedded machine learning practices solutions that enable a variety of real-time data-driven SHM applications. Finally, we wish to highlight that by using this specific Microcontroller Unit (MCU) and sensors, the proposed implementation aims for an overall low-cost data collection solution.

The dataset has been published as a dataset paper in 20th ACM Conference on Embedded Networked Sensor Systems (SenSys 2022), in the following workshop: The Fifth International Workshop on Data: Acquisition To Analysis (DATA '22).

The dataset is also available at https://github.com/Smart-Objects/Impact-Events-Dataset.

A validation assessment of Land Cover Monitoring, Assessment, and Projection Collection 1.0 annual land cover products (2000–2019) for Hawaii was conducted with an independently collected reference dataset. Reference data land cover attributes were assigned by trained interpreters for each year of the time series (2000–2019) to a reference sample of 600 Landsat resolution (30m x 30m) pixels. These pixels were selected from a sample frame of all pixels in the ARD grid system which fell within the map area (Dwyer et al., 2018). Interpretation used the TimeSync reference data collection tool which visualizes Landsat images and Landsat data values for all usable images in the time series (2000–2019) (Cohen et al., 2010). Interpreters also referred to air photos and high resolution images available in Google Earth as well as several ancillary data layers. The interpreted land cover attributes were crosswalked to the LCMAP annual land cover classes: Developed, Cropland, Grass/Shrub, Tree Cover, Wetland, Water, Snow/Ice and Barren. Validation analysis directly compared reference labels with annual LCMAP land cover map attributes by cross tabulation. The results of that assessment are reported here as confusion matrices for land cover agreement and land cover change agreement. Accuracy and standard errors have been calculated using stratified estimation (Stehman, 2014). Land cover class proportions were also estimated from the reference data for each year, 2000–2019. A cluster sampling formulation was used to calculate standard sampling error for summary tables reporting results for multiple years of data comparison.

This data set contains QA/QC-ed (Quality Assurance and Quality Control) water level data for the PLM1 and PLM6 wells. PLM1 and PLM6 are location identifiers used by the Watershed Function SFA project for two groundwater monitoring wells along an elevation gradient located along the lower montane life zone of a hillslope near the Pumphouse location at the East River Watershed, Colorado, USA. These wells are used to monitor subsurface water and carbon inventories and fluxes, and to determine the seasonally dependent flow of groundwater under the PLM hillslope. The downslope flow of groundwater in combination with data on groundwater chemistry (see related references) can be used to estimate rates of solute export from the hillslope to the floodplain and river. QA/QC analysis of measured groundwater levels in monitoring wells PLM-1 and PLM-6 included identification and flagging of duplicated values of timestamps, gap filling of missing timestamps and water levels, removal of abnormal/bad and outliers of measured water levels. The QA/QC analysis also tested the application of different QA/QC methods and the development of regular (5-minute, 1-hour, and 1-day) time series datasets, which can serve as a benchmark for testing other QA/QC techniques, and will be applicable for ecohydrological modeling. The package includes a Readme file, one R code file used to perform QA/QC, a series of 8 data csv files (six QA/QC-ed regular time series datasets of varying intervals (5-min, 1-hr, 1-day) and two files with QA/QC flagging of original data), and three files for the reporting format adoption of this dataset (InstallationMethods, file level metadata (flmd), and data dictionary (dd) files).QA/QC-ed data herein were derived from the original/raw data publication available at Williams et al., 2020 (DOI: 10.15485/1818367). For more information about running R code file (10.15485_1866836_QAQC_PLM1_PLM6.R) to reproduce QA/QC output files, see README (QAQC_PLM_readme.docx). This dataset replaces the previously published raw data time series, and is the final groundwater data product for the PLM wells in the East River. Complete metadata information on the PLM1 and PLM6 wells are available in a related dataset on ESS-DIVE: Varadharajan C, et al (2022). https://doi.org/10.15485/1660962. These data products are part of the Watershed Function Scientific Focus Area collection effort to further scientific understanding of biogeochemical dynamics from genome to watershed scales. 2022/09/09 Update: Converted data files using ESS-DIVE’s Hydrological Monitoring Reporting Format. With the adoption of this reporting format, the addition of three new files (v1_20220909_flmd.csv, V1_20220909_dd.csv, and InstallationMethods.csv) were added. The file-level metadata file (v1_20220909_flmd.csv) contains information specific to the files contained within the dataset. The data dictionary file (v1_20220909_dd.csv) contains definitions of column headers and other terms across the dataset. The installation methods file (InstallationMethods.csv) contains a description of methods associated with installation and deployment at PLM1 and PLM6 wells. Additionally, eight data files were re-formatted to follow the reporting format guidance (er_plm1_waterlevel_2016-2020.csv, er_plm1_waterlevel_1-hour_2016-2020.csv, er_plm1_waterlevel_daily_2016-2020.csv, QA_PLM1_Flagging.csv, er_plm6_waterlevel_2016-2020.csv, er_plm6_waterlevel_1-hour_2016-2020.csv, er_plm6_waterlevel_daily_2016-2020.csv, QA_PLM6_Flagging.csv). The major changes to the data files include the addition of header_rows above the data containing metadata about the particular well, units, and sensor description. 2023/01/18 Update: Dataset updated to include additional QA/QC-ed water level data up until 2022-10-12 for ER-PLM1 and 2022-10-13 for ER-PLM6. Reporting format specific files (v2_20230118_flmd.csv, v2_20230118_dd.csv, v2_20230118_InstallationMethods.csv) were updated to reflect the additional data. R code file (QAQC_PLM1_PLM6.R) was added to replace the previously uploaded HTML files to enable execution of the associated code. R code file (QAQC_PLM1_PLM6.R) and ReadMe file (QAQC_PLM_readme.docx) were revised to clarify where original data was retrieved from and to remove local file paths.

This is the supplementary data to the article "Large-scale winter catch crop monitoring with Sentinel-2 time series and machine learning–An alternative to on-site controls?" in Computers and Electronics in Agriculture by C. Schulz, A.-K. Holtgrave, and B. Kleinschmit 2021 (https://doi.org/xxxxxxxx). The data contains a zip-file with the following folders: data (parcels, filled and unfilled time series tables, feature extraction results and prediction results) (csv, shp), model (random forest models for catch crop prediction) (rds), and R (R script files for Random Forest model training and prediction with RStudio) (r).

The algorithms and RF models developed for this study were implemented via virtual Docker containers into the timeStamp software prototype which allows for large-scale automatized catch crop analysis on the parcel-level (timestamp.lup-umwelt.de). This software saves the raster data from the GTS² archive as parcel-wise clipped image time series into a PostGIS database. All further processing steps were performed with the statistical computing language R (RStudio Team, 2020). For raster data manipulation within the PostGIS database and downloading NDVI time series, we used the packages rpostgis (Bucklin and Basille, 2019) and RPostgreSQL (Conway et al., 2017). For time series filling and calculation of the predictors, we used the packages zoo (Zeileis et al., 2020), hydroGOF (Zambrano-Bigiarini, 2020), tsoutliers (de Lacalle, 2019), and changepoint (Killick et al., 2016). For RF modelling, we used the package caret (Kuhn et al., 2020).

This data set is a comma-separated values (CSV) file containing continuous hourly water quality observations of the Indian River in Sitka National Historical Park for monitoring years 2010-2021. Core parameters collected are temperature, dissolved oxygen, pH, and conductivity, obtained from multiparameter sondes during the ice-free season. Using the Aquarius Time-Series application, data have been quality controlled, graded against formal criteria specified in the protocol, drift corrected where appropriate, and certified for publication. The data set (CSV) and associated metadata are zipped into a site-specific archive (ZIP file), identified as the FQ_Q deliverable in the SEAN water quality protocol package FQ-2022.1, SOP 8.

The Barrow Strait flow observational program started in 1998 and continued for 13 years to 2012. This was followed by the Barrow Strait Real Time Observatory Project until 2016, afterwards the project is a combination of both projects and called Maritimes Region Barrow Strait Monitoring Program.

Barrow Strait is the widest of the 4 passages through the Canadian Archipelago making it an important monitoring location. The program explores the magnitude and variability of freshwater, heat and volume transports through the eastern Northwest Passage. The data measured included: profile CTD and time series temperature, salinity, density, currents, ice draft and ice drift velocity. The time series data collected was used to quantify the freshwater discharge from the Arctic Ocean through Barrow Strait into the northwest Atlantic, and link variability in this freshwater outflow to large scale weather patterns. Annual analysis performed on the collected data include: low-pass filtering, power spectra, progressive vector diagram, tidal analyses, seasonal and monthly averaged stats, mean flow, and ice velocities. Data from the program, along with a description of the methods used, have been published annually up to 2015 in the Canadian Data Report of Hydrography and Ocean Sciences report number 190 to 195, 173, 167, 166, 165, 161, and 157.

An ocean observatory was installed at the eastern end of the Northwest Passage in 2009 to provide hourly ocean and ice conditions for use by mariners and climate modellers. Data from instrumented moorings are acoustically transmitted to a "data hub" at the offshore end of an underwater cable, sent through the cable, and then transmitted from the shore station by Iridium satellite for access on the web (https://www.bio.gc.ca/science/newtech-technouvelles/observatory-observatoire-en.php)