Snapshot img

Data Science Platform Market Size 2025-2029

The data science platform market size is forecast to increase by USD 763.9 million at a CAGR of 40.2% between 2024 and 2029.

The market is experiencing significant growth, driven by the integration of artificial intelligence (AI) and machine learning (ML). This enhancement enables more advanced data analysis and prediction capabilities, making data science platforms an essential tool for businesses seeking to gain insights from their data. Another trend shaping the market is the emergence of containerization and microservices in platforms. This development offers increased flexibility and scalability, allowing organizations to efficiently manage their projects. 
However, the use of platforms also presents challenges, particularly In the area of data privacy and security. Ensuring the protection of sensitive data is crucial for businesses, and platforms must provide strong security measures to mitigate risks. In summary, the market is witnessing substantial growth due to the integration of AI and ML technologies, containerization, and microservices, while data privacy and security remain key challenges.

What will be the Size of the Data Science Platform Market During the Forecast Period?

Request Free Sample

The market is experiencing significant growth due to the increasing demand for advanced data analysis capabilities in various industries. Cloud-based solutions are gaining popularity as they offer scalability, flexibility, and cost savings. The market encompasses the entire project life cycle, from data acquisition and preparation to model development, training, and distribution. Big data, IoT, multimedia, machine data, consumer data, and business data are prime sources fueling this market's expansion. Unstructured data, previously challenging to process, is now being effectively managed through tools and software. Relational databases and machine learning models are integral components of platforms, enabling data exploration, preprocessing, and visualization.
Moreover, Artificial intelligence (AI) and machine learning (ML) technologies are essential for handling complex workflows, including data cleaning, model development, and model distribution. Data scientists benefit from these platforms by streamlining their tasks, improving productivity, and ensuring accurate and efficient model training. The market is expected to continue its growth trajectory as businesses increasingly recognize the value of data-driven insights.

How is this Data Science Platform Industry segmented and which is the largest segment?

The industry research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD million' for the period 2025-2029, as well as historical data from 2019-2023 for the following segments.

Deployment

  On-premises
  Cloud


Component

  Platform
  Services


End-user

  BFSI
  Retail and e-commerce
  Manufacturing
  Media and entertainment
  Others


Sector

  Large enterprises
  SMEs


Geography

  North America

    Canada
    US


  Europe

    Germany
    UK
    France


  APAC

    China
    India
    Japan


  South America

    Brazil


  Middle East and Africa

By Deployment Insights

The on-premises segment is estimated to witness significant growth during the forecast period.

On-premises deployment is a traditional method for implementing technology solutions within an organization. This approach involves purchasing software with a one-time license fee and a service contract. On-premises solutions offer enhanced security, as they keep user credentials and data within the company's premises. They can be customized to meet specific business requirements, allowing for quick adaptation. On-premises deployment eliminates the need for third-party providers to manage and secure data, ensuring data privacy and confidentiality. Additionally, it enables rapid and easy data access, and keeps IP addresses and data confidential. This deployment model is particularly beneficial for businesses dealing with sensitive data, such as those in manufacturing and large enterprises. While cloud-based solutions offer flexibility and cost savings, on-premises deployment remains a popular choice for organizations prioritizing data security and control.

Get a glance at the Data Science Platform Industry report of share of various segments. Request Free Sample

The on-premises segment was valued at USD 38.70 million in 2019 and showed a gradual increase during the forecast period.

Regional Analysis

North America is estimated to contribute 48% to the growth of the global market during the forecast period.

Technavio's analysts have elaborately explained the regional trends and drivers that shape the market during the forecast period.

For more insights on the market share of various regions, Request F

The Data Preparation Tools market is experiencing robust growth, projected to reach a market size of $3 billion in 2025 and exhibiting a Compound Annual Growth Rate (CAGR) of 17.7% from 2025 to 2033. This significant expansion is driven by several key factors. The increasing volume and velocity of data generated across industries necessitates efficient and effective data preparation processes to ensure data quality and usability for analytics and machine learning initiatives. The rising adoption of cloud-based solutions, coupled with the growing demand for self-service data preparation tools, is further fueling market growth. Businesses across various sectors, including IT and Telecom, Retail and E-commerce, BFSI (Banking, Financial Services, and Insurance), and Manufacturing, are actively seeking solutions to streamline their data pipelines and improve data governance. The diverse range of applications, from simple data cleansing to complex data transformation tasks, underscores the versatility and broad appeal of these tools. Leading vendors like Microsoft, Tableau, and Alteryx are continuously innovating and expanding their product offerings to meet the evolving needs of the market, fostering competition and driving further advancements in data preparation technology. This rapid growth is expected to continue, driven by ongoing digital transformation initiatives and the increasing reliance on data-driven decision-making. The segmentation of the market into self-service and data integration tools, alongside the varied applications across different industries, indicates a multifaceted and dynamic landscape. While challenges such as data security concerns and the need for skilled professionals exist, the overall market outlook remains positive, projecting substantial expansion throughout the forecast period. The adoption of advanced technologies like artificial intelligence (AI) and machine learning (ML) within data preparation tools promises to further automate and enhance the process, contributing to increased efficiency and reduced costs for businesses. The competitive landscape is dynamic, with established players alongside emerging innovators vying for market share, leading to continuous improvement and innovation within the industry.

Data Prep Market size was valued at USD 4.02 Billion in 2024 and is projected to reach USD 16.12 Billion by 2031, growing at a CAGR of 19% from 2024 to 2031.

Global Data Prep Market Drivers

Increasing Demand for Data Analytics: Businesses across all industries are increasingly relying on data-driven decision-making, necessitating the need for clean, reliable, and useful information. This rising reliance on data increases the demand for better data preparation technologies, which are required to transform raw data into meaningful insights.
Growing Volume and Complexity of Data: The increase in data generation continues unabated, with information streaming in from a variety of sources. This data frequently lacks consistency or organization, therefore effective data preparation is critical for accurate analysis. To assure quality and coherence while dealing with such a large and complicated data landscape, powerful technologies are required.
Increased Use of Self-Service Data Preparation Tools: User-friendly, self-service data preparation solutions are gaining popularity because they enable non-technical users to access, clean, and prepare data. independently. This democratizes data access, decreases reliance on IT departments, and speeds up the data analysis process, making data-driven insights more available to all business units.
Integration of AI and ML: Advanced data preparation technologies are progressively using AI and machine learning capabilities to improve their effectiveness. These technologies automate repetitive activities, detect data quality issues, and recommend data transformations, increasing productivity and accuracy. The use of AI and ML streamlines the data preparation process, making it faster and more reliable.
Regulatory Compliance Requirements: Many businesses are subject to tight regulations governing data security and privacy. Data preparation technologies play an important role in ensuring that data meets these compliance requirements. By giving functions that help manage and protect sensitive information these technologies help firms negotiate complex regulatory climates.
Cloud-based Data Management: The transition to cloud-based data storage and analytics platforms needs data preparation solutions that can work smoothly with cloud-based data sources. These solutions must be able to integrate with a variety of cloud settings to assist effective data administration and preparation while also supporting modern data infrastructure.

Benchmarking of Machine Learning Methods for Predicting Synthetic Lethality Interactions

Data Preparation and Download Instructions

After downloading data from this project, follow these steps to prepare the training data:

Step 1: Download all the data parts from the [](https://doi.org/10.5281/zenodo.13691648) provided in the repository.

Matters needing attention:

• The actual command will depend on how you're downloading files from Google Drive. For the convenience of downloading, the files have been compressed and split.
• We have prepared two versions of data, namely the complete version (`data_large.tar.gz`) and the version without PiLSL database (`data_small.tar.gz`).
• The decompressed size of the complete version is about 90GB, while the decompressed size of the version without PiLSL database is about 22GB.
• (Due to the extremely time-consuming database construction process, it is recommended to download the complete version of the data. However, the data version without PiLSL database can successfully run all 10 models except PiLSL.)

Step 2: Combine the parts into a single archive.

cat data_large.tar.gz.part* > data_large.tar.gz   # Complete version, the size after extracted is about 100GB.
# cat data_small.tar.gz.part* > data_small.tar.gz  # The version without PiLSL database, the size after extracted is about 25GB.

Step 3: Verify the integrity of the downloaded files.

md5sum -c data_large.tar.gz.md5
# md5sum -c data_small.tar.gz.md5 # The version without PiLSL database

Step 4: Extract the dataset.

tar -xzvf data_large.tar.gz
# tar -xzvf data_small.tar.gz # The version without PiLSL database

This model archive provides all data, code, and modeling results used in Barclay and others (2023) to assess the ability of process-guided deep learning stream temperature models to accurately incorporate groundwater-discharge processes. We assessed the performance of an existing process-guided deep learning stream temperature model of the Delaware River Basin (USA) and explored four approaches for improving groundwater process representation: 1) a custom loss function that leverages the unique patterns of air and water temperature coupling resulting from different temperature drivers, 2) inclusion of additional groundwater-relevant catchment attributes, 3) incorporation of additional process model outputs, and 4) a composite model. The associated manuscript examines changes in the predictive accuracy, feature importance, and predictive ability in un-seen reaches resulting from each of the four approaches. This model archive includes four zipped folders for 1) Data Preparation, 2) Model Code, 3) Model Predictions, and 4) the catchment attributes that were compiled for reaches in the study area. Instructions for running data preparation and modeling code can be found in the README.md files in 01_Data_Prep and 02_Model_Code respectively. File dictionaries have also been included and serve as metadata documentation for the files and datasets within the four zipped folders.

The included tests were performed at McMaster University in Hamilton, Ontario, Canada by Dr. Phillip Kollmeyer (phillip.kollmeyer@gmail.com). If this data is utilized for any purpose, it should be appropriately referenced. -A brand new 3Ah LG HG2 cell was tested in an 8 cu.ft. thermal chamber with a 75amp, 5 volt Digatron Firing Circuits Universal Battery Tester channel with a voltage and current accuracy of 0.1% of full scale. these data are used in the design process of an SOC estimator using a deep feedforward neural network (FNN) approach. The data also includes a description of data acquisition, data preparation, development of an FNN example script.

-Instructions for Downloading and Running the Script: 1-Select download all files from the Mendeley Data page (https://data.mendeley.com/datasets/cp3473x7xv/2).
2-The files will be downloaded as a zip file. Unzip the file to a folder, do not modify the folder structure.
3-Navigate to the folder with "FNN_xEV_Li_ion_SOC_EstimatorScript_March_2020.mlx" 4-Open and run "FNN_xEV_Li_ion_SOC_EstimatorScript_March_2020.mlx" 5-The matlab script should run without any modification, if there is an issue it's likely due to the testing and training data not being in the expected place. 6-The script is set by default to train for 50 epochs and to repeat the training 3 times. This should take 5-10 minutes to execute. 7-To recreate the results in the paper, set number of epochs to 5500 and number of repetitions to 10.

-The test data, or similar data, has been used for some publications, including: [1] C. Vidal, P. Kollmeyer, M. Naguib, P. Malysz, O. Gross, and A. Emadi, “Robust xEV Battery State-of-Charge Estimator Design using Deep Neural Networks,” in Proc WCX SAE World Congress Experience, Detroit, MI, Apr 2020 [2] C. Vidal, P. Kollmeyer, E. Chemali and A. Emadi, "Li-ion Battery State of Charge Estimation Using Long Short-Term Memory Recurrent Neural Network with Transfer Learning," 2019 IEEE Transportation Electrification Conference and Expo (ITEC), Detroit, MI, USA, 2019, pp. 1-6.

According to Cognitive Market Research, the global Ai Training Data market size is USD 1865.2 million in 2023 and will expand at a compound annual growth rate (CAGR) of 23.50% from 2023 to 2030.

The demand for Ai Training Data is rising due to the rising demand for labelled data and diversification of AI applications.
Demand for Image/Video remains higher in the Ai Training Data market.
The Healthcare category held the highest Ai Training Data market revenue share in 2023.
North American Ai Training Data will continue to lead, whereas the Asia-Pacific Ai Training Data market will experience the most substantial growth until 2030.

Market Dynamics of AI Training Data Market

Key Drivers of AI Training Data Market

Rising Demand for Industry-Specific Datasets to Provide Viable Market Output

A key driver in the AI Training Data market is the escalating demand for industry-specific datasets. As businesses across sectors increasingly adopt AI applications, the need for highly specialized and domain-specific training data becomes critical. Industries such as healthcare, finance, and automotive require datasets that reflect the nuances and complexities unique to their domains. This demand fuels the growth of providers offering curated datasets tailored to specific industries, ensuring that AI models are trained with relevant and representative data, leading to enhanced performance and accuracy in diverse applications.

In July 2021, Amazon and Hugging Face, a provider of open-source natural language processing (NLP) technologies, have collaborated. The objective of this partnership was to accelerate the deployment of sophisticated NLP capabilities while making it easier for businesses to use cutting-edge machine-learning models. Following this partnership, Hugging Face will suggest Amazon Web Services as a cloud service provider for its clients.

(Source: about:blank)

Advancements in Data Labelling Technologies to Propel Market Growth

The continuous advancements in data labelling technologies serve as another significant driver for the AI Training Data market. Efficient and accurate labelling is essential for training robust AI models. Innovations in automated and semi-automated labelling tools, leveraging techniques like computer vision and natural language processing, streamline the data annotation process. These technologies not only improve the speed and scalability of dataset preparation but also contribute to the overall quality and consistency of labelled data. The adoption of advanced labelling solutions addresses industry challenges related to data annotation, driving the market forward amidst the increasing demand for high-quality training data.

In June 2021, Scale AI and MIT Media Lab, a Massachusetts Institute of Technology research centre, began working together. To help doctors treat patients more effectively, this cooperation attempted to utilize ML in healthcare.

www.ncbi.nlm.nih.gov/pmc/articles/PMC7325854/

Restraint Factors Of AI Training Data Market

Data Privacy and Security Concerns to Restrict Market Growth

A significant restraint in the AI Training Data market is the growing concern over data privacy and security. As the demand for diverse and expansive datasets rises, so does the need for sensitive information. However, the collection and utilization of personal or proprietary data raise ethical and privacy issues. Companies and data providers face challenges in ensuring compliance with regulations and safeguarding against unauthorized access or misuse of sensitive information. Addressing these concerns becomes imperative to gain user trust and navigate the evolving landscape of data protection laws, which, in turn, poses a restraint on the smooth progression of the AI Training Data market.

How did COVID–19 impact the Ai Training Data market?

The COVID-19 pandemic has had a multifaceted impact on the AI Training Data market. While the demand for AI solutions has accelerated across industries, the availability and collection of training data faced challenges. The pandemic disrupted traditional data collection methods, leading to a slowdown in the generation of labeled datasets due to restrictions on physical operations. Simultaneously, the surge in remote work and the increased reliance on AI-driven technologies for various applications fueled the need for diverse and relevant training data. This duali...

Machine Learning pipeline used to provide toxicity prediction in FunTox-Networks

01_DATA # preprocessing and filtering of raw activity data from ChEMBL
- Chembl_v25 # latest activity assay data set from ChEMBL (retrieved Nov 2019)
- filt_stats.R # Filtering and preparation of raw data
- Filtered # output data sets from filt_stats.R
- toxicity_direction.csv # table of toxicity measurements and their proportionality to toxicity

02_MolDesc # Calculation of molecular descriptors for all compounds within the filtered ChEMBL data set
- datastore # files with all compounds and their calculated molecular descriptors based on SMILES
- scripts
- calc_molDesc.py # calculates for all compounds based on their smiles the molecular descriptors
- chemopy-1.1 # used python package for descriptor calculation as decsribed in: https://doi.org/10.1093/bioinformatics/btt105

03_Averages # Calculation of moving averages for levels and organisms as required for calculation of Z-scores
- datastore # output files with statistics calculated by make_Z.R
- scripts
-make_Z.R # script to calculate statistics to calculate Z-scores as used by the regression models

04_ZScores # Calculation of Z-scores and preparation of table to fit regression models
- datastore # Z-normalized activity data and molecular descriptors in the form as used for fitting regression models
- scripts
-calc_Ztable.py # based on activity data, molecular descriptors and Z-statistics, the learning data is calculated

05_Regression # Performing regression. Preparation of data by removing of outliers based on a linear regression model. Learning of random forest regression models. Validation of learning process by cross validation and tuning of hyperparameters.

- datastore # storage of all random forest regression models and average level of Z output value per level and organism (zexp_*.tsv)
- scripts
- data_preperation.R # set up of regression data set, removal of outliers and optional removal of fields and descriptors
- Rforest_CV.R # analysis of machine learning by cross validation, importance of regression variables and tuning of hyperparameters (number of trees, split of variables)
- Rforest.R # based on analysis of Rforest_CV.R learning of final models

rregrs_output
# early analysis of regression model performance with the package RRegrs as described in: https://doi.org/10.1186/s13321-015-0094-2

This data release and model archive provides all data, code, and modelling results used in Topp et al. (2023) to examine the influence of deep learning architecture on generalizability when predicting stream temperature in the Delaware River Basin (DRB). Briefly, we modeled stream temperature in the DRB using two spatially and temporally aware process guided deep learning models (a recurrent graph convolution network - RGCN, and a temporal convolution graph model - Graph WaveNet). The associated manuscript explores how the architectural differences between the two models influence how they learn spatial and temporal relationships, and how those learned relationships influence a model's ability to accurately predict stream temperature as domains shift towards out-of-bounds conditions. This data release and model archive contains three zipped folders for 1) Data Preparation, 2) Modelling Code, and 3) Model Predictions. Instructions for running data preparation code and modelling code can be found in the README.md files in 01_Data_Prep and 02_Model_Code respectively.

Abstract This dataset and its metadata statement were supplied to the Bioregional Assessment Programme by a third party and are presented here as originally supplied. This is Version 2 of the …Show full descriptionAbstract This dataset and its metadata statement were supplied to the Bioregional Assessment Programme by a third party and are presented here as originally supplied. This is Version 2 of the Australian Soil Depth of Regolith product of the Soil and Landscape Grid of Australia (produced 2015-06-01). The Soil and Landscape Grid of Australia has produced a range of digital soil attribute products. The digital soil attribute maps are in raster format at a resolution of 3 arc sec (~90 x 90 m pixels). Attribute Definition: The regolith is the in situ and transported material overlying unweathered bedrock; Units: metres; Spatial prediction method: data mining using piecewise linear regression; Period (temporal coverage; approximately): 1900-2013; Spatial resolution: 3 arc seconds (approx 90m); Total number of gridded maps for this attribute:3; Number of pixels with coverage per layer: 2007M (49200 * 40800); Total size before compression: about 8GB; Total size after compression: about 4GB; Data license : Creative Commons Attribution 3.0 (CC By); Variance explained (cross-validation): R^2 = 0.38; Target data standard: GlobalSoilMap specifications; Format: GeoTIFF. Dataset History The methodology consisted of the following steps: (i) drillhole data preparation, (ii) compilation and selection of the environmental covariate raster layers and (iii) model implementation and evaluation. Drillhole data preparation: Drillhole data was sourced from the National Groundwater Information System (NGIS) database. This spatial database holds nationally consistent information about bores that were drilled as part of the Bore Construction Licensing Framework (http://www.bom.gov.au/water/groundwater/ngis/). The database contains 357,834 bore locations with associated lithology, bore construction and hydrostratigraphy records. This information was loaded into a relational database to facilitate analysis. Regolith depth extraction: The first step was to recognise and extract the boundary between the regolith and bedrock within each drillhole record. This was done using a key word look-up table of bedrock or lithology related words from the record descriptions. 1,910 unique descriptors were discovered. Using this list of new standardised terms analysis of the drillholes was conducted, and the depth value associated with the word in the description that was unequivocally pointing to reaching fresh bedrock material was extracted from each record using a tool developed in C# code. The second step of regolith depth extraction involved removal of drillhole bedrock depth records deemed necessary because of the "noisiness" in depth records resulting from inconsistencies we found in drilling and description standards indentified in the legacy database. On completion of the filtering and removal of outliers the drillhole database used in the model comprised of 128,033 depth sites. Selection and preparation of environmental covariates The environmental correlations style of DSM applies environmental covariate datasets to predict target variables, here regolith depth. Strongly performing environmental covariates operate as proxies for the factors that control regolith formation including climate, relief, parent material organisms and time (Jenny, 1941 Depth modelling was implemented using the PC-based R-statistical software (R Core Team, 2014), and relied on the R-Cubist package (Kuhn et al. 2013). To generate modelling uncertainty estimates, the following procedures were followed: (i) the random withholding of a subset comprising 20% of the whole depth record dataset for external validation; (ii) Bootstrap sampling 100 times of the remaining dataset to produce repeated model training datasets, each time. The Cubist model was then run repeated times to produce a unique rule set for each of these training sets. Repeated model runs using different training sets, a procedure referred to as bagging or bootstrap aggregating, is a machine learning ensemble procedure designed to improve the stability and accuracy of the model. The Cubist rule sets generated were then evaluated and applied spatially calculating a mean predicted value (i.e. the final map). The 5% and 95% confidence intervals were estimated for each grid cell (pixel) in the prediction dataset by combining the variance from the bootstrapping process and the variance of the model residuals. Version 2 differs from version 1, in that the modelling of depths was performed on the log scale to better conform to assumptions of normality used in calculating the confidence intervals. The method to estimate the confidence intervals was improved to better represent the full range of variability in the modelling process. (Wilford et al, in press) Dataset Citation CSIRO (2015) AUS Soil and Landscape Grid National Soil Attribute Maps - Depth of Regolith (3" resolution) - Release 2. Bioregional Assessment Source Dataset. Viewed 22 June 2018, http://data.bioregionalassessments.gov.au/dataset/c28597e8-8cfc-4b4f-8777-c9934051cce2.

This is Version 2 of the Depth of Regolith product of the Soil and Landscape Grid of Australia (produced 2015-06-01).

The Soil and Landscape Grid of Australia has produced a range of digital soil attribute products. The digital soil attribute maps are in raster format at a resolution of 3 arc sec (~90 x 90 m pixels).

Attribute Definition: The regolith is the in situ and transported material overlying unweathered bedrock; Units: metres; Spatial prediction method: data mining using piecewise linear regression; Period (temporal coverage; approximately): 1900-2013; Spatial resolution: 3 arc seconds (approx 90m); Total number of gridded maps for this attribute:3; Number of pixels with coverage per layer: 2007M (49200 * 40800); Total size before compression: about 8GB; Total size after compression: about 4GB; Data license : Creative Commons Attribution 4.0 (CC BY); Variance explained (cross-validation): R^2 = 0.38; Target data standard: GlobalSoilMap specifications; Format: GeoTIFF. Lineage: The methodology consisted of the following steps: (i) drillhole data preparation, (ii) compilation and selection of the environmental covariate raster layers and (iii) model implementation and evaluation.

Drillhole data preparation: Drillhole data was sourced from the National Groundwater Information System (NGIS) database. This spatial database holds nationally consistent information about bores that were drilled as part of the Bore Construction Licensing Framework (http://www.bom.gov.au/water/groundwater/ngis/). The database contains 357,834 bore locations with associated lithology, bore construction and hydrostratigraphy records. This information was loaded into a relational database to facilitate analysis.

Regolith depth extraction: The first step was to recognise and extract the boundary between the regolith and bedrock within each drillhole record. This was done using a key word look-up table of bedrock or lithology related words from the record descriptions. 1,910 unique descriptors were discovered. Using this list of new standardised terms analysis of the drillholes was conducted, and the depth value associated with the word in the description that was unequivocally pointing to reaching fresh bedrock material was extracted from each record using a tool developed in C# code.

The second step of regolith depth extraction involved removal of drillhole bedrock depth records deemed necessary because of the “noisiness” in depth records resulting from inconsistencies we found in drilling and description standards indentified in the legacy database.

On completion of the filtering and removal of outliers the drillhole database used in the model comprised of 128,033 depth sites.

Selection and preparation of environmental covariates The environmental correlations style of DSM applies environmental covariate datasets to predict target variables, here regolith depth. Strongly performing environmental covariates operate as proxies for the factors that control regolith formation including climate, relief, parent material organisms and time.

Depth modelling was implemented using the PC-based R-statistical software (R Core Team, 2014), and relied on the R-Cubist package (Kuhn et al. 2013). To generate modelling uncertainty estimates, the following procedures were followed: (i) the random withholding of a subset comprising 20% of the whole depth record dataset for external validation; (ii) Bootstrap sampling 100 times of the remaining dataset to produce repeated model training datasets, each time. The Cubist model was then run repeated times to produce a unique rule set for each of these training sets. Repeated model runs using different training sets, a procedure referred to as bagging or bootstrap aggregating, is a machine learning ensemble procedure designed to improve the stability and accuracy of the model. The Cubist rule sets generated were then evaluated and applied spatially calculating a mean predicted value (i.e. the final map). The 5% and 95% confidence intervals were estimated for each grid cell (pixel) in the prediction dataset by combining the variance from the bootstrapping process and the variance of the model residuals. Version 2 differs from version 1, in that the modelling of depths was performed on the log scale to better conform to assumptions of normality used in calculating the confidence intervals. The method to estimate the confidence intervals was improved to better represent the full range of variability in the modelling process. (Wilford et al, in press)

Example values of selected features.

RMSE results of CO predictions by using LSTM with different activation functions.

Electrochromic devices, capable of modulating light transmittance under the influence of an electric field, have garnered significant interest in the field of smart windows and car rearview mirrors. However, the development of high-performance electrochromic devices via large-scale explorations under miscellaneous experimental settings remains challenging and is still an urgent problem to be solved. In this study, we employed a two-step machine learning approach, combining machine learning algorithms such as KNN and XGBoost with the reality of electrochromic devices, to construct a comprehensive evaluation system for electrochromic materials. Utilizing our predictive evaluation system, we successfully screened the preparation conditions for the best-performing device, which was experimentally verified to have a high transmittance modulation amplitude (62.6%) and fast response time (5.7 s/7.1 s) at 70 A/m2. To test its stability, experiments over a long cycle time (1000 cycles) are performed. In this study, we develop an innovative framework for assessing the performance of electrochromic material devices. Our approach effectively filters experimental samples based on their distinct properties, substantially minimizing the expenditure of human and material resources in electrochromic research. Our approach to a mathematical machine learning evaluation framework for device performance has effectively propelled and informed research in electrochromic devices.

Effective management of non-indigenous species requires knowledge of their dispersal factors and founder events. We aim to identify the main environmental drivers favouring dispersal events along the invasion gradient and to characterize the spatial patterns of genetic diversity in feral populations of the non-native pink salmon within its epicentre of invasion in Norway. We first conducted SDM using four modelling techniques with varying levels of complexity, which encompassed both regression-based and tree-based machine-learning algorithms, using climatic data from the present to 2050. Then we used the triple-enzyme restriction-site associated DNA sequencing (3RADseq) approach to genotype over 30,000 high-quality single-nucleotide polymorphisms to elucidate patterns of genetic diversity and gene flow within the pink salmon putative invasion hotspot. We discovered temperature- and precipitation-related variables drove pink salmon distributional shifts across its non-native ranges, and ..., 3RAD library preparation and sequencing: We prepared RADseq libraries using the Adapterama III library preparation protocol of Bayona-Vásquez et al., (2019; their Supplemental File SI). For each sample, ~40-100 ng of genomic DNA were digested for 1 h at 37 °C in a solution with 1.5 µl of 10x Cutsmart® buffer, 0.25 µl (NEB®) of Read 1 enzyme (MspI) at 20 U/µl, 0.25 µl of Read 2 enzyme (BamHI-HF) at 20 U/µl, 0.25 µl of Read 1 adapter dimer-cutting enzyme (ClaI) at 20 U/ µl, 1 µl of i5Tru adapter at 2.5 µM, 1 µl of i7Tru adapter at 2.5 µM and 0.75 µl of dH2O. After digestion/ligation, samples were pooled and cleaned with 1.2x Sera-Mag SpeedBeads (Fisher Scientiifc™) in a 1.2:1 (SpeedBeads:DNA) ratio, and we eluted cleaned DNA in 60 µL of TLE. An enrichment PCR of each sample was carried with 10 µl of 5x Kapa Long Range Buffer (Kapa Biosystems, Inc.), 0.25 µl of KAPA LongRange DNA Polymerase at 5 U/µl, 1.5 µl of dNTPs mix (10 mM each dNTP), 3.5 µl of MgCl2 at 25 mM, 2.5 µl of iTru5 primer at ..., , # Genome-wide SNP datasets for the non-native pink salmon in Norway

The complete single nucleotide polymorphisms (SNPs) dataset underwent several filtering steps, including thinning SNPs to a density of one SNP per kilobase, removing closely related individuals, eliminating candidate paralogous regions of the genome (known as multi-site variants or MSVs), and excluding SNPs within non-chromosomal scaffolds. This resulted in a final panel of 43,719 polymorphic SNPs, with a genotyping rate of 0.98 and a sample size of 73 individuals. We eliminated all SNPs that were potentially influenced by selection, as determined by two genome scans for outlier tests. As a result, we obtained a final dataset consisting of 33,860 SNPs that were considered to be neutral. From this dataset, we derived a SNP subset dataset of 250 'diagnostic' SNPs with the highest locus-specific *F*ST.

Description of the data and file structure

The ‘neutral’ full-SNP dataset: OGO-3RAD-D2-NEUTRAL-SNPS.vcf **NOTE:*...

Proteomic fingerprinting using MALDI-TOF mass spectrometry is a well-established tool for identifying microorganisms and has shown promising results for identification of animal species, particularly disease vectors and marine organisms. However, few studies have tested species identification across different orders and classes. In this study, we collected data from 1,246 specimens and 198 species to test species identification in a diverse dataset. We also evaluated different specimen preparation and data processing approaches for machine learning and developed a workflow to optimize classification using random forest. Our results showed high success rates of over 90%, but we also found that the size of the reference library affects classification error. Additionally, we demonstrated the ability of the method to differentiate marine cryptic-species complexes and to distinguish sexes within species. Methods Tissue for measurements was taken mainly from the marine organisms tissue bank of the Senckenberg am Meer, German Centre for Marine Biodiversity Research, which was established using samples from numerous studies (Knebelsberger and Thiel, 2014; Knebelsberger et al., 2014; Markert et al., 2014; Gebhardt and Knebelsberger, 2015; Raupach et al., 2015; Barco et al., 2016; Laakmann et al., 2016; Rossel et al., 2020b) (supplementary table S1 for accession numbers) on North Sea metazoans. The material from this collection was taken from specimens processed for COI-barcoding to create reference libraries for a variety of marine animal groups. During this process, tissue samples of the respective specimens were stored in ethanol at -80°C. Tissue samples were available for Bivalvia (muscle, 18 species), Cephalopoda (muscle from arm, 12 species), Gastropoda (muscle from foot, 24 species), Polyplacophora (muscle from foot, 2 species), Ascidiacea (tissue, 1 species), Teleostei (muscle, 67 species), Elasmobranchii (muscle, 7 species), Malacostraca (muscle from foot or chelae, 39 species), Thecostraca (muscle from foot, 1 species), Pycnogonida (leg fragment, 1 species), Asteroidea (tube feet, 10 species), Ophiuroidea (tissue from arm, 10 species) and Echinoidea (tissue from the base of the tubercle, 6 species) (nspecies= 198, nspecimens=1,246). Sample preparation The basic protocol of sample preparation was the same for all analyzed tissue samples. A very small tissue fragment (< 1 mm3) was incubated for 5 minutes in α-cyano-4-hydroxycinnamic acid (HCCA) as a saturated solution in 50% acetonitrile, 47.5% molecular grade water and 2.5% trifluoroacetic acid. Tissue from crustacean Cancer pagurus Linnaeus, 1758, the fish Clupea harengus Linnaeus, 1758, the cephalopod Eledone cirrhosa (Lamarck, 1798) and the echinoderm Stichastrella rosea (O.F. Müller, 1776) was used to find an optimal tissue to HCCA matrix ratio. Tissue was weighted on a METTLER TOLEDO XS3DU micro-balance and the amount of matrix was adjusted to tissue weight to obtain the desired ratios ranging from 0.012 µg µl-1 to 200 µg µl-1. After incubation, 1.5 µl of the solution was transferred to 10 spots on a target plate, respectively. Mass spectra were measured with a Microflex LT/SH System (Bruker Daltonics) using method MBTAuto. Peak evaluation was carried out in a mass peak range between 2 k – 10 k Dalton (Da) using a centroid peak detection algorithm, a signal to noise threshold of 2 and a minimum intensity threshold of 600. To create a sum spectrum, 160 satisfactory shots were summed up. Resulting from observations during this initial test, a fast applicable protocol was developed without the need to weigh each tissue sample. Matrix volume was added to tissue samples depending on tissue volume, i.e. tissue samples were always completely covered by HCCA matrix with a small layer (ca. 1 mm) of supernatant. Samples were incubated for 5 minutes and 1.5 µl of the solution were transferred to a single spot on a target plate for measurement. Each spot was measured between two to three times.

Distribution fitting results of cash out (CO) prediction errors.

Sharing cooking recipes is a great way to exchange culinary ideas and provide instructions for food preparation. However, categorizing raw recipes found online into appropriate food genres can be challenging due to a lack of adequate labeled data. In this study, we present a dataset named the “Assorted, Archetypal, and Annotated Two Million Extended (3A2M+) Cooking Recipe Dataset” that contains two million culinary recipes labeled in respective categories with extended named entities extracted from recipe descriptions. This collection of data includes various features such as title, NER, directions, and extended NER, as well as nine different labels representing genres including bakery, drinks, non-veg, vegetables, fast food, cereals, meals, sides, and fusions. The proposed pipeline named 3A2M+ extends the size of the Named Entity Recognition (NER) list to address missing named entities like heat, time or process from the recipe directions using two NER extraction tools. 3A2M+ dataset provides a comprehensive solution to the various challenging recipe-related tasks, including classification, named entity recognition, and recipe generation. Furthermore, we have demonstrated traditional machine learning, deep learning and pre-trained language models to classify the recipes into their corresponding genre and achieved an overall accuracy of 98.6%. Our investigation indicates that the title feature played a more significant role in classifying the genre.

This is the companion dataset to publication {TBD}. It contains 1) seasonal composites of predicted maize cover and yield at 10 m resolution in Rwanda for two annual agricultural seasons over five years, 2) scripts for the end-to-end machine learning pipeline that produces these data products, and 3) data or references needed as inputs to the pipeline.

1) Maize cover and yield seasonal composites

The data are provided here as netCDF4 files with four dimensions for x, y, band, and season. They can also be accessed as Google Earth ImageCollections at:

• https://code.earthengine.google.com/?asset=projects/b2p-geospatial/assets/lulc_classifier_composite
• https://code.earthengine.google.com/?asset=projects/b2p-geospatial/assets/maize_yield_composite

Land cover and maize classification

The land cover classification file is found at data/composites/lulc_classifier_Rwanda_2019to2023.nc.

The land cover classification images contain 3 bands/variables: maizeProb, the raw predicted probability of the pixel being maize given by the gradient boosted tree model; majorityClass, the categorical land cover class with the highest predicted probability among any of the nine classes in the respective pixel; and optimalClass, the categorical land cover class adjusted to agree with national statistics for expected maize area.

The land cover classes map to the raster values as follows:

{
  1: 'maize',
  2: 'nonmaize_annual',
  3: 'nonmaize_perennial',
  4: 'scrub_shrub_land',
  5: 'forest',
  6: 'flooded_vegetation',
  7: 'water',
  8: 'structure',
  9: 'bare'
}

The dataset includes 5 years (2019-2023) and 10 seasons - the available time period at time of publication. In Rwanda, maize is typically planted and harvested during two distinct agricultural seasons per year: Season A from September to February and Season B from March to June. Therefore the seasons in the data are: 2019_Season_A, 2019_Season_B, 2020_Season_A, 2020_Season_B, 2021_Season_A, 2021_Season_B, 2022_Season_A, 2022_Season_B, 2023_Season_A, 2023_Season_B.

Maize yield

The maize yield file is found at data/composites/maize_yield_Rwanda_2019to2023.nc.

Each of the images in the yield composites has 3 bands/variables also: maizeYield, the model's output of continuous predicted yield (kg/ha) in each pixel regardless of land class; maizeYield_majorityClass, predicted maize yield masked to the majority class land classification; and maizeYieldAdj_optimalClass, where the raw predicted yields were masked to the optimal maize classification land cover layer and normalized to national statistics.

The dataset includes the same seasons as the classification product; see above for a description.

2) End-to-end machine learning pipeline

All earth observation imagery, analysis, and outputs unless otherwise stated were hosted in the Google Earth Engine (GEE) environment and developed with the Earth Engine Python API in Python v3.10. To set up a local conda environment use the scripts/environment.yml file. The user must have Google Cloud Storage (GCS) and Google Earth Engine (GEE) accounts. The pipeline, at this scale, will incur some processing and storage fees, although Google offers a free trial to all new users and the total cost of the high-resolution wall-to-wall predictions is nominal (~$20 for one season).

The scripts needed to perform the pipeline are located in the scripts folder.

The files contained in the scripts/helpers directory will be called by various subsequent scripts and do not to be run interactively by the user.

Follow the script in the order described below. The user should pause after running each script and confirm that all outputs were created and loaded to GCS before continuing the pipeline; for some steps this may take hours to days depending on processing speed.

Google Cloud Storage and Earth Engine set-up

Users should specify the names of the bucket and asset project that were chosen during set up of their GCS and GEE environments in the Objects section of scripts/helpers/maize_pipeline_0_workspace.py.

Pipeline set-up

In scripts/pipeline_setup, you will find the following scripts to perform data preparation of inputs into model building and prediction.

• maize_pipeline_1_clean_training_data.py - Cleans and merges all available crop label and yield data for model training and validation
• maize_pipeline_2_dwnld_data_training.py - Downloads satellite-derived and auxiliary features at training data points for model building
• maize_pipeline_3_dwnld_data_inference.py - Downloads satellite-derived and auxiliary features at every 10 m pixel in Rwanda on a district-wise basis for prediction

Land cover and maize classification

In scripts/maize_classification, you will find the following scripts to perform model building, prediction, and post-processing for the classificaton of land cover type and maize cover.

• maize_classifier_1_feature_selection.py - Selects features subset for land cover classification with mutual information score or variable importance
• maize_classifier_2_build_model.py - Builds gradient boosted tree model for land cover classification from training data
• maize_classifier_3_prediction.py - Applies model for land cover classification to every 10 m pixel in Rwanda by season and district
• maize_classifier_4_postprocess.py - Mosaics district-wise predictions and normalizes maize cover predictions to national agricultural statistics

Maize yield

In scripts/maize_yield, you will find the following scripts to perform modeling building, prediction, and post-processing for maize yield estimation.

• maize_yield_1_build_model.py - Builds gradient boosted tree model and performs bias correction for maize yield estimation from training data
• maize_yield_2_prediction.py - Applies model for maize yield estimation to every 10 m pixel in Rwanda by season and district
• maize_yield_3_postprocess.py - Mosaics district-wise predictions and normalizes maize yield predictions to national agricultural statistics

If you are running the entire pipeline with refreshed training data and model building, run each of these scripts, in order. By default, the script will run all A and B seasons from 2019A to current. Otherwise, if you just wish to re-run or update seasonal predictions from the existing classification or yield model run maize_pipeline_3_dwnld_data_inference.py to download the seasonal feature data across Rwanda and maize_classifier_3_prediction.pyand maize_classifier_4_postprocess.py for classification predictions or maize_yield_2_prediction.py and maize_yield_3_postprocess.py for yield predictions, making sure to specify which season(s) are of interest in each script. However to do this, you also need to have a copy of the previously built models in your GCS (provided at data/models).

3) Input data into machine learning pipeline

A description of datasets that must be sourced outside of the GEE platform is provided below. When available, the primary data source is also included in the directory data/baselayers. All other data, including Sentinel-2 imagery, auxiliary data, and other existing global land cover classificaiton products are hosted on GEE and called by the scripts directly. All datasets last accessed on 12 March 2024.

Administrative and geological boundaries

• World Countries - Downloaded from The World Bank Official Boundaries and included here at data/baselayers/World_Countries.
• Rwanda district boundaries - Downloaded from The World Bank Rwanda Admin Boundaries And Villages and included here at data/baselayers/WB_NISR_2018. This should be loaded into a FeatureCollection GEE asset named districts_fc for use in the pipeline.
• Rwanda agro-ecological zones - Downloaded from Nzeyimana, Hartemink & Geissen (2016) and included here at data/baselayers/MINAGRI_AEZ_1980. This should be loaded into a FeatureCollection GEE asset named aez_rwanda for use in the pipeline.

Global land cover classification product

• Microsoft/Impact Observatory LULC - Although the 10m Annual Land Use Land Cover (9-class) V1 product contains data from 2017-2022, only the LULC map from the year 2021 was used, provided here at data/baselayers/impactobs_lulc_rwa_2021.tif. This should be loaded into an ImageCollection GEE asset named impact_obs_lulc for use in the pipeline.

(The others - Dynamic World and ESA's WorldCover - are hosted on GEE directly.)

Land cover labels and maize yield crop cuttings

• One Acre Fund - Contact authors to request access as this dataset is not hosted publicly.
• RTI International - The original source of

Passive Acoustic Monitoring (PAM) is emerging as a solution for monitoring species and environmental change over large spatial and temporal scales. However, drawing rigorous conclusions based on acoustic recordings is challenging, as there is no consensus over which approaches, and indices are best suited for characterizing marine and terrestrial acoustic environments. Here, we describe the application of multiple machine-learning techniques to the analysis of a large PAM dataset. We combine pre-trained acoustic classification models (VGGish, NOAA & Google Humpback Whale Detector), dimensionality reduction (UMAP), and balanced random forest algorithms to demonstrate how machine-learned acoustic features capture different aspects of the marine environment. The UMAP dimensions derived from VGGish acoustic features exhibited good performance in separating marine mammal vocalizations according to species and locations. RF models trained on the acoustic features performed well for labelled sounds in the 8 kHz range, however, low and high-frequency sounds could not be classified using this approach. The workflow presented here shows how acoustic feature extraction, visualization, and analysis allow for establishing a link between ecologically relevant information and PAM recordings at multiple scales. The datasets and scripts provided in this repository allow replicating the results presented in the publication. Methods Data acquisition and preparation We collected all records available in the Watkins Marine Mammal Database website listed under the “all cuts'' page. For each audio file in the WMD the associated metadata included a label for the sound sources present in the recording (biological, anthropogenic, and environmental), as well as information related to the location and date of recording. To minimize the presence of unwanted sounds in the samples, we only retained audio files with a single source listed in the metadata. We then labelled the selected audio clips according to taxonomic group (Odontocetae, Mysticetae), and species. We limited the analysis to 12 marine mammal species by discarding data when a species: had less than 60 s of audio available, had a vocal repertoire extending beyond the resolution of the acoustic classification model (VGGish), or was recorded in a single country. To determine if a species was suited for analysis using VGGish, we inspected the Mel-spectrograms of 3-s audio samples and only retained species with vocalizations that could be captured in the Mel-spectrogram (Appendix S1). The vocalizations of species that produce very low frequency, or very high frequency were not captured by the Mel-spectrogram, thus we removed them from the analysis. To ensure that records included the vocalizations of multiple individuals for each species, we only considered species with records from two or more different countries. Lastly, to avoid overrepresentation of sperm whale vocalizations, we excluded 30,000 sperm whale recordings collected in the Dominican Republic. The resulting dataset consisted in 19,682 audio clips with a duration of 960 milliseconds each (0.96 s) (Table 1). The Placentia Bay Database (PBD) includes recordings collected by Fisheries and Oceans Canada in Placentia Bay (Newfoundland, Canada), in 2019. The dataset consisted of two months of continuous recordings (1230 hours), starting on July 1st, 2019, and ending on August 31st 2029. The data was collected using an AMAR G4 hydrophone (sensitivity: -165.02 dB re 1V/µPa at 250 Hz) deployed at 64 m of depth. The hydrophone was set to operate following 15 min cycles, with the first 60 s sampled at 512 kHz, and the remaining 14 min sampled at 64 kHz. For the purpose of this study, we limited the analysis to the 64 kHz recordings. Acoustic feature extraction The audio files from the WMD and PBD databases were used as input for VGGish (Abu-El-Haija et al., 2016; Chung et al., 2018), a CNN developed and trained to perform general acoustic classification. VGGish was trained on the Youtube8M dataset, containing more than two million user-labelled audio-video files. Rather than focusing on the final output of the model (i.e., the assigned labels), here the model was used as a feature extractor (Sethi et al., 2020). VGGish converts audio input into a semantically meaningful vector consisting of 128 features. The model returns features at multiple resolution: ~1 s (960 ms); ~5 s (4800 ms); ~1 min (59’520 ms); ~5 min (299’520 ms). All of the visualizations and results pertaining to the WMD were prepared using the finest feature resolution of ~1 s. The visualizations and results pertaining to the PBD were prepared using the ~5 s features for the humpback whale detection example, and were then averaged to an interval of 30 min in order to match the temporal resolution of the environmental measures available for the area. UMAP ordination and visualization UMAP is a non-linear dimensionality reduction algorithm based on the concept of topological data analysis which, unlike other dimensionality reduction techniques (e.g., tSNE), preserves both the local and global structure of multivariate datasets (McInnes et al., 2018). To allow for data visualization and to reduce the 128 features to two dimensions for further analysis, we applied Uniform Manifold Approximation and Projection (UMAP) to both datasets and inspected the resulting plots. The UMAP algorithm generates a low-dimensional representation of a multivariate dataset while maintaining the relationships between points in the global dataset structure (i.e., the 128 features extracted from VGGish). Each point in a UMAP plot in this paper represents an audio sample with duration of ~ 1 second (WMD dataset), ~ 5 seconds (PBD dataset, humpback whale detections), or 30 minutes (PBD dataset, environmental variables). Each point in the two-dimensional UMAP space also represents a vector of 128 VGGish features. The nearer two points are in the plot space, the nearer the two points are in the 128-dimensional space, and thus the distance between two points in UMAP reflects the degree of similarity between two audio samples in our datasets. Areas with a high density of samples in UMAP space should, therefore, contain sounds with similar characteristics, and such similarity should decrease with increasing point distance. Previous studies illustrated how VGGish and UMAP can be applied to the analysis of terrestrial acoustic datasets (Heath et al., 2021; Sethi et al., 2020). The visualizations and classification trials presented here illustrate how the two techniques (VGGish and UMAP) can be used together for marine ecoacoustics analysis. UMAP visualizations were prepared the umap-learn package for Python programming language (version 3.10). All UMAP visualizations presented in this study were generated using the algorithm’s default parameters.
Labelling sound sources The labels for the WMD records (i.e., taxonomic group, species, location) were obtained from the database metadata. For the PBD recordings, we obtained measures of wind speed, surface temperature, and current speed from (Fig 1) an oceanographic buy located in proximity of the recorder. We choose these three variables for their different contributions to background noise in marine environments. Wind speed contributes to underwater background noise at multiple frequencies, ranging 500 Hz to 20 kHz (Hildebrand et al., 2021). Sea surface temperature contributes to background noise at frequencies between 63 Hz and 125 Hz (Ainslie et al., 2021), while ocean currents contribute to ambient noise at frequencies below 50 Hz (Han et al., 2021) Prior to analysis, we categorized the environmental variables and assigned the categories as labels to the acoustic features (Table 2). Humpback whale vocalizations in the PBD recordings were processed using the humpback whale acoustic detector created by NOAA and Google (Allen et al., 2021), providing a model score for every ~5 s sample. This model was trained on a large dataset (14 years and 13 locations) using humpback whale recordings annotated by experts (Allen et al., 2021). The model returns scores ranging from 0 to 1 indicating the confidence in the predicted humpback whale presence. We used the results of this detection model to label the PBD samples according to presence of humpback whale vocalizations. To verify the model results, we inspected all audio files that contained a 5 s sample with a model score higher than 0.9 for the month of July. If the presence of a humpback whale was confirmed, we labelled the segment as a model detection. We labelled any additional humpback whale vocalization present in the inspected audio files as a visual detection, while we labelled other sources and background noise samples as absences. In total, we labelled 4.6 hours of recordings. We reserved the recordings collected in August to test the precision of the final predictive model. Label prediction performance We used Balanced Random Forest models (BRF) provided in the imbalanced-learn python package (Lemaître et al., 2017) to predict humpback whale presence and environmental conditions from the acoustic features generated by VGGish. We choose BRF as the algorithm as it is suited for datasets characterized by class imbalance. The BRF algorithm performs under sampling of the majority class prior to prediction, allowing to overcome class imbalance (Lemaître et al., 2017). For each model run, the PBD dataset was split into training (80%) and testing (20%) sets. The training datasets were used to fine-tune the models though a nested k-fold cross validation approach with ten-folds in the outer loop, and five-folds in the inner loop. We selected nested cross validation as it allows optimizing model hyperparameters and performing model evaluation in a single step. We used the default parameters of the BRF algorithm, except for the ‘n_estimators’ hyperparameter, for which we tested