Synthetic Data Generation Engine Market Outlook

According to our latest research, the global synthetic data generation engine market size reached USD 1.48 billion in 2024. The market is experiencing robust expansion, driven by the increasing demand for privacy-compliant data and advanced analytics solutions. The market is projected to grow at a remarkable CAGR of 35.6% from 2025 to 2033, reaching an estimated USD 18.67 billion by the end of the forecast period. This rapid growth is primarily propelled by the adoption of artificial intelligence (AI) and machine learning (ML) across various industry verticals, along with the escalating need for high-quality, diverse datasets that do not compromise sensitive information.

One of the primary growth factors fueling the synthetic data generation engine market is the heightened focus on data privacy and regulatory compliance. With stringent regulations such as GDPR, CCPA, and HIPAA being enforced globally, organizations are increasingly seeking solutions that enable them to generate and utilize data without exposing real customer information. Synthetic data generation engines provide a powerful means to create realistic, anonymized datasets that retain the statistical properties of original data, thus supporting robust analytics and model development while ensuring compliance with data protection laws. This capability is especially critical for sectors like healthcare, banking, and government, where data sensitivity is paramount.

Another significant driver is the surging adoption of AI and ML models across industries, which require vast volumes of diverse and representative data for training and validation. Traditional data collection methods often fall short due to limitations in data availability, quality, or privacy concerns. Synthetic data generation engines address these challenges by enabling the creation of customized datasets tailored for specific use cases, including rare-event modeling, edge-case scenario testing, and data augmentation. This not only accelerates innovation but also reduces the time and cost associated with data acquisition and labeling, making it a strategic asset for organizations seeking to maintain a competitive edge in AI-driven markets.

Moreover, the increasing integration of synthetic data generation engines into enterprise IT ecosystems is being catalyzed by advancements in cloud computing and scalable software architectures. Cloud-based deployment models are making these solutions more accessible and cost-effective for organizations of all sizes, from startups to large enterprises. The flexibility to generate, store, and manage synthetic datasets in the cloud enhances collaboration, speeds up development cycles, and supports global operations. As a result, cloud adoption is expected to further accelerate market growth, particularly among businesses undergoing digital transformation and seeking to leverage synthetic data for innovation and compliance.

Regionally, North America currently dominates the synthetic data generation engine market, accounting for the largest revenue share in 2024, followed closely by Europe and the Asia Pacific. North America's leadership is attributed to the presence of major technology providers, robust regulatory frameworks, and a high level of AI adoption across industries. Europe is experiencing rapid growth due to strong data privacy regulations and a thriving technology ecosystem, while Asia Pacific is emerging as a lucrative market, driven by digitalization initiatives and increasing investments in AI and analytics. The regional outlook suggests that market expansion will be broad-based, with significant opportunities for vendors and stakeholders across all major geographies.

Component Analysis

The component segment of the synthetic data generation engine market is bifurcated into software and services, each playing a vital role in the overall ecosystem. Software solutions form the backbone of this market, providing the core algorithms and platforms that enable the generation, management, and deployment of synthetic datasets. These platforms are continually evolving, integrating advanced techniques such as generative adversarial networks (GANs), variational autoencoders, and other deep learning models to produce highly realistic and diverse synthetic data. The software segment is anticipated to maintain its dominance throughout the forecast period, as organizations increasingly invest in proprietary and commercial tools to address their un

The performance of statistical methods is frequently evaluated by means of simulation studies. In case of network meta-analysis of binary data, however, available data- generating models are restricted to either inclusion of two-armed trials or the fixed-effect model. Based on data-generation in the pairwise case, we propose a framework for the simulation of random-effect network meta-analyses including multi-arm trials with binary outcome. The only of the common data-generating models which is directly applicable to a random-effects network setting uses strongly restrictive assumptions. To overcome these limitations, we modify this approach and derive a related simulation procedure using odds ratios as effect measure. The performance of this procedure is evaluated with synthetic data and in an empirical example.

The use of synthetic data is recognized as a crucial step in the development of neural network-based Artificial Intelligence (AI) systems. While the methods for generating synthetic data for AI applications in other domains have a role in certain biomedical AI systems, primarily related to image processing, there is a critical gap in the generation of time series data for AI tasks where it is necessary to know how the system works. This is most pronounced in the ability to generate synthetic multi-dimensional molecular time series data (subsequently referred to as synthetic mediator trajectories or SMTs); this is the type of data that underpins research into biomarkers and mediator signatures for forecasting various diseases and is an essential component of the drug development pipeline. We argue the insufficiency of statistical and data-centric machine learning (ML) means of generating this type of synthetic data is due to a combination of factors: perpetual data sparsity due to the Curse of Dimensionality, the inapplicability of the Central Limit Theorem in terms of making assumptions about the statistical distributions of this type of data, and the inability to use ab initio simulations due to the state of perpetual epistemic incompleteness in cellular/molecular biology. Alternatively, we present a rationale for using complex multi-scale mechanism-based simulation models, constructed and operated on to account for perpetual epistemic incompleteness and the need to provide maximal expansiveness in concordance with the Maximal Entropy Principle. These procedures provide for the generation of SMT that minimizes the known shortcomings associated with neural network AI systems, namely overfitting and lack of generalizability. The generation of synthetic data that accounts for the identified factors of multi-dimensional time series data is an essential capability for the development of mediator-biomarker based AI forecasting systems, and therapeutic control development and optimization.

These are simulated data without any identifying information or informative birth-level covariates. We also standardize the pollution exposures on each week by subtracting off the median exposure amount on a given week and dividing by the interquartile range (IQR) (as in the actual application to the true NC birth records data). The dataset that we provide includes weekly average pregnancy exposures that have already been standardized in this way while the medians and IQRs are not given. This further protects identifiability of the spatial locations used in the analysis. This dataset is not publicly accessible because: EPA cannot release personally identifiable information regarding living individuals, according to the Privacy Act and the Freedom of Information Act (FOIA). This dataset contains information about human research subjects. Because there is potential to identify individual participants and disclose personal information, either alone or in combination with other datasets, individual level data are not appropriate to post for public access. Restricted access may be granted to authorized persons by contacting the party listed. It can be accessed through the following means: File format: R workspace file; “Simulated_Dataset.RData”. Metadata (including data dictionary) • y: Vector of binary responses (1: adverse outcome, 0: control) • x: Matrix of covariates; one row for each simulated individual • z: Matrix of standardized pollution exposures • n: Number of simulated individuals • m: Number of exposure time periods (e.g., weeks of pregnancy) • p: Number of columns in the covariate design matrix • alpha_true: Vector of “true” critical window locations/magnitudes (i.e., the ground truth that we want to estimate) Code Abstract We provide R statistical software code (“CWVS_LMC.txt”) to fit the linear model of coregionalization (LMC) version of the Critical Window Variable Selection (CWVS) method developed in the manuscript. We also provide R code (“Results_Summary.txt”) to summarize/plot the estimated critical windows and posterior marginal inclusion probabilities. Description “CWVS_LMC.txt”: This code is delivered to the user in the form of a .txt file that contains R statistical software code. Once the “Simulated_Dataset.RData” workspace has been loaded into R, the text in the file can be used to identify/estimate critical windows of susceptibility and posterior marginal inclusion probabilities. “Results_Summary.txt”: This code is also delivered to the user in the form of a .txt file that contains R statistical software code. Once the “CWVS_LMC.txt” code is applied to the simulated dataset and the program has completed, this code can be used to summarize and plot the identified/estimated critical windows and posterior marginal inclusion probabilities (similar to the plots shown in the manuscript). Optional Information (complete as necessary) Required R packages: • For running “CWVS_LMC.txt”: • msm: Sampling from the truncated normal distribution • mnormt: Sampling from the multivariate normal distribution • BayesLogit: Sampling from the Polya-Gamma distribution • For running “Results_Summary.txt”: • plotrix: Plotting the posterior means and credible intervals Instructions for Use Reproducibility (Mandatory) What can be reproduced: The data and code can be used to identify/estimate critical windows from one of the actual simulated datasets generated under setting E4 from the presented simulation study. How to use the information: • Load the “Simulated_Dataset.RData” workspace • Run the code contained in “CWVS_LMC.txt” • Once the “CWVS_LMC.txt” code is complete, run “Results_Summary.txt”. Format: Below is the replication procedure for the attached data set for the portion of the analyses using a simulated data set: Data The data used in the application section of the manuscript consist of geocoded birth records from the North Carolina State Center for Health Statistics, 2005-2008. In the simulation study section of the manuscript, we simulate synthetic data that closely match some of the key features of the birth certificate data while maintaining confidentiality of any actual pregnant women. Availability Due to the highly sensitive and identifying information contained in the birth certificate data (including latitude/longitude and address of residence at delivery), we are unable to make the data from the application section publically available. However, we will make one of the simulated datasets available for any reader interested in applying the method to realistic simulated birth records data. This will also allow the user to become familiar with the required inputs of the model, how the data should be structured, and what type of output is obtained. While we cannot provide the application data here, access to the North Carolina birth records can be requested through the North Carolina State Center for Health Statistics, and requires an appropriate data use agreement. Description Permissions: These are simulated data without any identifying information or informative birth-level covariates. We also standardize the pollution exposures on each week by subtracting off the median exposure amount on a given week and dividing by the interquartile range (IQR) (as in the actual application to the true NC birth records data). The dataset that we provide includes weekly average pregnancy exposures that have already been standardized in this way while the medians and IQRs are not given. This further protects identifiability of the spatial locations used in the analysis. This dataset is associated with the following publication: Warren, J., W. Kong, T. Luben, and H. Chang. Critical Window Variable Selection: Estimating the Impact of Air Pollution on Very Preterm Birth. Biostatistics. Oxford University Press, OXFORD, UK, 1-30, (2019).

Supplementary files for article A genetically-optimised artificial life algorithm for complexity-based synthetic dataset generation

Algorithmic evaluation is a vital step in developing new approaches to machine learning and relies on the availability of existing datasets. However, real-world datasets often do not cover the necessary complexity space required to understand an algorithm’s domains of competence. As such, the generation of synthetic datasets to fill gaps in the complexity space has gained attention, offering a means of evaluating algorithms when data is unavailable. Existing approaches to complexity-focused data generation are limited in their ability to generate solutions that invoke similar classification behaviour to real data. The present work proposes a novel method (Sy:Boid) for complexity-based synthetic data generation, adapting and extending the Boid algorithm that was originally intended for computer graphics simulations. Sy:Boid embeds the modified Boid algorithm within an evolutionary multi-objective optimisation algorithm to generate synthetic datasets which satisfy predefined magnitudes of complexity measures. Sy:Boid is evaluated and compared to labelling-based and sampling-based approaches to data generation to understand its ability to generate a wide variety of realistic datasets. Results demonstrate Sy:Boid is capable of generating datasets across a greater portion of the complexity space than existing approaches. Furthermore, the produced datasets were observed to invoke very similar classification behaviours to that of real data.

Quantum-AI Synthetic Data Generator Market Outlook

According to our latest research, the global Quantum-AI Synthetic Data Generator market size reached USD 1.98 billion in 2024, reflecting robust momentum driven by the convergence of quantum computing and artificial intelligence technologies in data generation. The market is experiencing a significant compound annual growth rate (CAGR) of 32.1% from 2025 to 2033. At this pace, the market is forecasted to reach USD 24.8 billion by 2033. This remarkable growth is propelled by the escalating demand for high-quality synthetic data across industries to enhance AI model training, ensure data privacy, and overcome data scarcity challenges.

One of the primary growth drivers for the Quantum-AI Synthetic Data Generator market is the increasing reliance on advanced machine learning and deep learning models that require vast amounts of diverse, high-fidelity data. Traditional data sources often fall short in volume, variety, and compliance with privacy regulations. Quantum-AI synthetic data generators address these challenges by producing realistic, representative datasets that mimic real-world scenarios without exposing sensitive information. This capability is particularly crucial in regulated sectors such as healthcare and finance, where data privacy and security are paramount. As organizations seek to accelerate AI adoption while minimizing ethical and legal risks, the demand for sophisticated synthetic data solutions continues to rise.

Another significant factor fueling market expansion is the rapid evolution of quantum computing and its integration with AI algorithms. Quantum computing’s superior processing power enables the generation of complex, large-scale datasets at unprecedented speeds and accuracy. This synergy allows enterprises to simulate intricate data patterns and rare events that would be difficult or impossible to capture through conventional means. Additionally, the proliferation of AI-driven applications in sectors like autonomous vehicles, predictive maintenance, and personalized medicine is amplifying the need for synthetic data generators that can support advanced analytics and model validation. The ongoing advancements in quantum hardware, coupled with the growing ecosystem of AI tools, are expected to further catalyze innovation and adoption in this market.

Moreover, the shift toward digital transformation and the growing adoption of cloud-based solutions are reshaping the landscape of the Quantum-AI Synthetic Data Generator market. Enterprises of all sizes are embracing synthetic data generation to streamline data workflows, reduce operational costs, and accelerate time-to-market for AI-powered products and services. Cloud deployment models offer scalability, flexibility, and seamless integration with existing data infrastructure, making synthetic data generation accessible even to resource-constrained organizations. As digital ecosystems evolve and data-driven decision-making becomes a competitive imperative, the strategic importance of synthetic data generation is set to intensify, fostering sustained market growth through 2033.

From a regional perspective, North America currently leads the market, driven by early technology adoption, substantial investments in quantum and AI research, and a vibrant ecosystem of startups and established technology firms. Europe follows closely, benefiting from strong regulatory frameworks and robust funding for AI innovation. The Asia Pacific region is witnessing the fastest growth, fueled by expanding digital economies, government initiatives supporting AI and quantum technology, and increasing awareness of synthetic data’s strategic value. As global enterprises seek to harness the power of quantum-AI synthetic data generators to gain a competitive edge, regional dynamics will continue to shape market trajectories and opportunities.

Component Analysis

The Component segment of the Quantum-AI Synthetic Data Generator

TiCaM Synthectic Images: A Time-of-Flight In-Car Cabin Monitoring Dataset is a time-of-flight dataset of car in-cabin images providing means to test extensive car cabin monitoring systems based on deep learning methods. The authors provide a synthetic image dataset of car cabin images similar to the real dataset leveraging advanced simulation software’s capability to generate abundant data with little effort. This can be used to test domain adaptation between synthetic and real data for select classes. For both datasets the authors provide ground truth annotations for 2D and 3D object detection, as well as for instance segmentation.

The Generative AI Market size was valued at USD 16.88 billion in 2023 and is projected to reach USD 149.04 billion by 2032, exhibiting a CAGR of 36.5 % during the forecasts period. The generative AI market specifically means the segment of a market that sells products based on the AI technologies for creating content that includes text, images, audio content, and videos. While generative AI models are mainly based on machine learning, especially neural networks, it synthesises new content that is similar to human-generated data. Some of them are as follows- Creation of contents and designs, more specifically in discovery of any drug and through customized marketing strategies. It is applied to areas including, but not limited to entertainment, health care, and finances. Modern developments indicate the emergence of AI-art, AI-music, and AI-writings, the usage of generative AI for automated communication with customers, and the enhancement of AI-ethics and -regulations. Challenges are defined by the constant enhancements in AI algorithms and the rising need for automation and inventiveness in various fields. Recent developments include: In April 2023, Microsoft Corp. collaborated with Epic Systems, an American healthcare software company, to incorporate large language model tools and AI into Epic’s electronic health record software. This partnership aims to use generative AI to help healthcare providers increase productivity while reducing administrative burden , In March 2021, MOSTLY AI Inc. announced its partnership with Erste Group, an Australian bank to provide its AI-based synthetic data solution. Using synthetic data, Erste Group aims to boost its digital banking innovation and enable data-based development .

The Synthetic Patient Data in OMOP Dataset is a synthetic database released by the Centers for Medicare and Medicaid Services (CMS) Medicare Claims Synthetic Public Use Files (SynPUF). It is synthetic data containing 2008-2010 Medicare insurance claims for development and demonstration purposes. It has been converted to the Observational Medical Outcomes Partnership (OMOP) common data model from its original form, CSV, by the open source community as released on GitHub Please refer to the CMS Linkable 2008–2010 Medicare Data Entrepreneurs’ Synthetic Public Use File (DE-SynPUF) User Manual for details regarding how DE-SynPUF was created." This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery .

This is Part 2/2 of the ActiveHuman dataset! Part 1 can be found here. Dataset Description ActiveHuman was generated using Unity's Perception package. It consists of 175428 RGB images and their semantic segmentation counterparts taken at different environments, lighting conditions, camera distances and angles. In total, the dataset contains images for 8 environments, 33 humans, 4 lighting conditions, 7 camera distances (1m-4m) and 36 camera angles (0-360 at 10-degree intervals). The dataset does not include images at every single combination of available camera distances and angles, since for some values the camera would collide with another object or go outside the confines of an environment. As a result, some combinations of camera distances and angles do not exist in the dataset. Alongside each image, 2D Bounding Box, 3D Bounding Box and Keypoint ground truth annotations are also generated via the use of Labelers and are stored as a JSON-based dataset. These Labelers are scripts that are responsible for capturing ground truth annotations for each captured image or frame. Keypoint annotations follow the COCO format defined by the COCO keypoint annotation template offered in the perception package.

Folder configuration The dataset consists of 3 folders:

JSON Data: Contains all the generated JSON files. RGB Images: Contains the generated RGB images. Semantic Segmentation Images: Contains the generated semantic segmentation images.

Essential Terminology

Annotation: Recorded data describing a single capture. Capture: One completed rendering process of a Unity sensor which stored the rendered result to data files (e.g. PNG, JPG, etc.). Ego: Object or person on which a collection of sensors is attached to (e.g., if a drone has a camera attached to it, the drone would be the ego and the camera would be the sensor). Ego coordinate system: Coordinates with respect to the ego. Global coordinate system: Coordinates with respect to the global origin in Unity. Sensor: Device that captures the dataset (in this instance the sensor is a camera). Sensor coordinate system: Coordinates with respect to the sensor. Sequence: Time-ordered series of captures. This is very useful for video capture where the time-order relationship of two captures is vital. UIID: Universal Unique Identifier. It is a unique hexadecimal identifier that can represent an individual instance of a capture, ego, sensor, annotation, labeled object or keypoint, or keypoint template.

Dataset Data The dataset includes 4 types of JSON annotation files files:

annotation_definitions.json: Contains annotation definitions for all of the active Labelers of the simulation stored in an array. Each entry consists of a collection of key-value pairs which describe a particular type of annotation and contain information about that specific annotation describing how its data should be mapped back to labels or objects in the scene. Each entry contains the following key-value pairs:

id: Integer identifier of the annotation's definition. name: Annotation name (e.g., keypoints, bounding box, bounding box 3D, semantic segmentation). description: Description of the annotation's specifications. format: Format of the file containing the annotation specifications (e.g., json, PNG). spec: Format-specific specifications for the annotation values generated by each Labeler.

Most Labelers generate different annotation specifications in the spec key-value pair:

BoundingBox2DLabeler/BoundingBox3DLabeler:

label_id: Integer identifier of a label. label_name: String identifier of a label. KeypointLabeler:

template_id: Keypoint template UUID. template_name: Name of the keypoint template. key_points: Array containing all the joints defined by the keypoint template. This array includes the key-value pairs:

label: Joint label. index: Joint index. color: RGBA values of the keypoint. color_code: Hex color code of the keypoint skeleton: Array containing all the skeleton connections defined by the keypoint template. Each skeleton connection defines a connection between two different joints. This array includes the key-value pairs:

label1: Label of the first joint. label2: Label of the second joint. joint1: Index of the first joint. joint2: Index of the second joint. color: RGBA values of the connection. color_code: Hex color code of the connection. SemanticSegmentationLabeler:

label_name: String identifier of a label. pixel_value: RGBA values of the label. color_code: Hex color code of the label.

captures_xyz.json: Each of these files contains an array of ground truth annotations generated by each active Labeler for each capture separately, as well as extra metadata that describe the state of each active sensor that is present in the scene. Each array entry in the contains the following key-value pairs:

id: UUID of the capture. sequence_id: UUID of the sequence. step: Index of the capture within a sequence. timestamp: Timestamp (in ms) since the beginning of a sequence. sensor: Properties of the sensor. This entry contains a collection with the following key-value pairs:

sensor_id: Sensor UUID. ego_id: Ego UUID. modality: Modality of the sensor (e.g., camera, radar). translation: 3D vector that describes the sensor's position (in meters) with respect to the global coordinate system. rotation: Quaternion variable that describes the sensor's orientation with respect to the ego coordinate system. camera_intrinsic: matrix containing (if it exists) the camera's intrinsic calibration. projection: Projection type used by the camera (e.g., orthographic, perspective). ego: Attributes of the ego. This entry contains a collection with the following key-value pairs:

ego_id: Ego UUID. translation: 3D vector that describes the ego's position (in meters) with respect to the global coordinate system. rotation: Quaternion variable containing the ego's orientation. velocity: 3D vector containing the ego's velocity (in meters per second). acceleration: 3D vector containing the ego's acceleration (in ). format: Format of the file captured by the sensor (e.g., PNG, JPG). annotations: Key-value pair collections, one for each active Labeler. These key-value pairs are as follows:

id: Annotation UUID . annotation_definition: Integer identifier of the annotation's definition. filename: Name of the file generated by the Labeler. This entry is only present for Labelers that generate an image. values: List of key-value pairs containing annotation data for the current Labeler.

Each Labeler generates different annotation specifications in the values key-value pair:

BoundingBox2DLabeler:

label_id: Integer identifier of a label. label_name: String identifier of a label. instance_id: UUID of one instance of an object. Each object with the same label that is visible on the same capture has different instance_id values. x: Position of the 2D bounding box on the X axis. y: Position of the 2D bounding box position on the Y axis. width: Width of the 2D bounding box. height: Height of the 2D bounding box. BoundingBox3DLabeler:

label_id: Integer identifier of a label. label_name: String identifier of a label. instance_id: UUID of one instance of an object. Each object with the same label that is visible on the same capture has different instance_id values. translation: 3D vector containing the location of the center of the 3D bounding box with respect to the sensor coordinate system (in meters). size: 3D vector containing the size of the 3D bounding box (in meters) rotation: Quaternion variable containing the orientation of the 3D bounding box. velocity: 3D vector containing the velocity of the 3D bounding box (in meters per second). acceleration: 3D vector containing the acceleration of the 3D bounding box acceleration (in ). KeypointLabeler:

label_id: Integer identifier of a label. instance_id: UUID of one instance of a joint. Keypoints with the same joint label that are visible on the same capture have different instance_id values. template_id: UUID of the keypoint template. pose: Pose label for that particular capture. keypoints: Array containing the properties of each keypoint. Each keypoint that exists in the keypoint template file is one element of the array. Each entry's contents have as follows:

index: Index of the keypoint in the keypoint template file. x: Pixel coordinates of the keypoint on the X axis. y: Pixel coordinates of the keypoint on the Y axis. state: State of the keypoint.

The SemanticSegmentationLabeler does not contain a values list.

egos.json: Contains collections of key-value pairs for each ego. These include:

id: UUID of the ego. description: Description of the ego. sensors.json: Contains collections of key-value pairs for all sensors of the simulation. These include:

id: UUID of the sensor. ego_id: UUID of the ego on which the sensor is attached. modality: Modality of the sensor (e.g., camera, radar, sonar). description: Description of the sensor (e.g., camera, radar).

Image names The RGB and semantic segmentation images share the same image naming convention. However, the semantic segmentation images also contain the string Semantic_ at the beginning of their filenames. Each RGB image is named "e_h_l_d_r.jpg", where:

e denotes the id of the environment. h denotes the id of the person. l denotes the id of the lighting condition. d denotes the camera distance at which the image was captured. r denotes the camera angle at which the image was captured.

PLEASE NOTE: This is a Synthetic data file, also known as a Dummy File - it is NOT real data. This synthetic data file should not be used for purposes other than to develop and test computer programs that are to be submitted by remote access. Each record in the synthetic file matches the format and content parameters of the real Statistics Canada Master File with which it is associated, but the data themselves have been 'made up'. They do NOT represent responses from real individuals and should NOT be used for actual analysis. These data are provided solely for the purpose of testing statistical packing 'code' (e.g. SPSS syntax, SAS programs, etc.) in preparation for analysis using the associated Master File in a Research Data Centre, by Remote Job Submission, or by some other means of secure access. If statistical analysis 'code' works with the synthetic data, researchers can have some confidence that the same code will run successfully against the Master File data in the Research Data Centres. The Canadian Community Health Survey (CCHS) is a cross-sectional survey that collects information related to health status, health care utilization and health determinants for the Canadian population. It surveys a large sample of respondents and is designed to provide reliable estimates at the health region level. In 2007, major changes were made to the CCHS design. Data is now collected on an ongoing basis with annual releases, rather than every two years as was the case prior to 2007. The survey's objectives were also revised and are as follows: • support health surveillance programs by providing health data at the national, provincial and intra-provincial levels; • provide a single data source for health research on small populations and rare characteristics; • timely release of information easily accessible to a diverse community of users; and • create a flexible survey instrument that includes a rapid response option to address emerging issues related to the health of the population. The CCHS data is always collected from persons aged 12 and over living in private dwellings in the 115 health regions covering all provinces and territories. Excluded from the sampling frame are individuals living on Indian Reserves and on Crown Lands, institutional residents, full-time members of the Canadian Forces, and residents of certain remote regions. The CCHS covers approximately 98% of the Canadian population aged 12 and over. The CCHS produces three types of microdata files: master files; share files; and public use microdata files (PUMF). The characteristics of each of these files are presented in the User Guide. The PUMF is released every two years and contains two years of data.

Synthetic data for assessing and comparing local post-hoc explanation of detected process shift

DOI

10.5281/zenodo.15000635

Synthetic dataset contains data used in experiment described in article submitted to Computers in Industry journal entitled

Assessing and Comparing Local Post-hoc Explanation for Shift Detection in Process Monitoring.

The citation will be updated immediately after the article will be accepted.

Particular data.mat files are stored in a subfolder structure, which clearly assigns the particular file to

on of the tested cases.

For example, data for experiments with normally distributed data, known number of shifted variables and 5 variables are stored in path ormal\known_number\5_vars\rho0.1.

The meaning of particular folders is explained here:

normal - all variables are normally distributed

not-normal - copula based multivariate distribution based on normal and gamma marginal distributions and defined correlation

known_number - known number of shifted variables (methods used this information, which is not available in real world)

unknown_number - unknown number of shifted variables, realistic case

2_vars - data with 2 variables (n=2)

...

10_vars - data with 10 variables (n=2)

rho0.1 - correlation among all variables is 0.1

...

rho0.9 - correlation among all variables is 0.9

Each data.mat file contains the following variables:

LIME_res nval x n results of LIME explanation

MYT_res nval x n results of MYT explanation

NN_res nval x n results of ANN explanation

X p x 11000 Unshifted data

S n x n sigma matrix (covariance matrix) for the unshifted data

mu 1xn mean parameter for the unshifted data

n 1x1 number of variables (dimensionality)

trn_set n x ntrn x 2 train set for ANN explainer,

         trn_set(:,:,1) are values of variables from shifted process

         trn_set(:,:,2) labels denoting which variables are shifted 

         trn_set(i,j,2) is 1 if ith variable of jth sample trn_set(:,j,1) is shifted

val_set n x 95 x 2 validation set used for testing and generating LIME_res, MYT_res and NN_res

The SIDTD dataset is an extension of the MIDV2020 dataset. Initially, the MIDV2020 dataset is composed of forged ID documents, as all documents are generated by means of AI techniques. These generated documents are considered in the SIDTD dataset as representative of bona fide. On the other hand, the documents generated are considered as being forged versions of them. The corpus of the dataset is composed by ten European nationalities that are equally represented: Albanian, Azerbaijani, Estonian, Finnish, Greek, Lithuanian, Russian, Serbian, Slovakian, and Spanish. We employ two techniques for generating composite PAIs: Crop & Replace and inpainting. Datase contains videos, and clips, of captured ID Documents with different backgrounds, we add the same type of data for the forged ID Document images generated using the techniques described. The protocol employed to generate the dataset is as follows: We printed 191 counterfeit ID documents on paper using an HP Color LaserJet E65050 printer. Then, the documents were laminated with 100-micron-thick laminating pouches to enhance realism and manually cropped. CVC’s employees were requested to use their smartphones to record videos of forged ID documents from SIDTD. This approach aimed to capture a diverse range of video qualities, backgrounds, durations, and light intensities

Please note: This is a Synthetic data file, also known as a Dummy File - it is NOT real data. This synthetic data file should not be used for purposes other than to develop and test computer programs that are to be submitted by remote access. Each record in the synthetic file matches the format and content parameters of the real Statistics Canada Master File with which it is associated, but the data themselves have been 'made up'. They do NOT represent responses from real individuals and should NOT be used for actual analysis. These data are provided solely for the purpose of testing statistical packing 'code' (e.g. SPSS syntax, SAS programs, etc.) in preparation for analysis using the associated Master File in a Research Data Centre, by Remote Job Submission, or by some other means of secure access. If statistical analysis 'code' works with the synthetic data, researchers can have some confidence that the same code will run successfully against the Master File data in the Research Data Centres. The Canadian Community Health Survey (CCHS) is a cross-sectional survey that collects information related to health status, health care utilization and health determinants for the Canadian population. Starting in 2007, the CCHS now operates using continuous collection. It is a large sample, general population health survey, designed to provide reliable estimates at the health region level. In order to provide researchers with a means to access the master file(s), a remote access facility has been implemented. Remote access provides researchers with the possibility to submit computer programs via e-mail to a dedicated address (cchs-escc@statcan.ca), and to receive the results by return e-mail. To obtain remote access privileges, it is necessary that researchers obtain advance approval from the Health Statistics Division. Requests must be submitted to the aforementioned e-mail address and must provide the following, clearly itemized information: •the researcher’s affiliation, • the name of all researchers involved in the project, • the title of the research project, • an abstract of the project, • the goals of the research, • the data to which access is required (survey, cycle), • why the project requires access to the master data rather than the PUMF, • why Remote Access service is chosen rather the on-site access in a Research Data Centre (RDC), • the expected results, and • the project’s expected completion date. Further information is available by contacting the CCHS team at the above e-mail address or by phone at (613) 951-1653. Once the request for remote access has been approved, the researcher can submit his/her computer programs to the CCHS team for processing on the master file(s). The computer output is reviewed by the team for confidentiality concerns and returned to the researcher. However, the correctness and accuracy of each program submission remains, at all times, the sole responsibility of the researcher.

Objective: Although many clinical metrics are associated with proximity to decompensation in heart failure (HF), none are individually accurate enough to risk-stratify HF patients on a patient-by-patient basis. The dire consequences of this inaccuracy in risk stratification have profoundly lowered the clinical threshold for application of high-risk surgical intervention, such as ventricular assist device placement. Machine learning can detect non-intuitive classifier patterns that allow for innovative combination of patient feature predictive capability. A machine learning-based clinical tool to identify proximity to catastrophic HF deterioration on a patient-specific basis would enable more efficient direction of high-risk surgical intervention to those patients who have the most to gain from it, while sparing others. Synthetic electronic health record (EHR) data are statistically indistinguishable from the original protected health information, and can be analyzed as if they were original data but without any privacy concerns. We demonstrate that synthetic EHR data can be easily accessed and analyzed and are amenable to machine learning analyses.Methods: We developed synthetic data from EHR data of 26,575 HF patients admitted to a single institution during the decade ending on 12/31/2018. Twenty-seven clinically-relevant features were synthesized and utilized in supervised deep learning and machine learning algorithms (i.e., deep neural networks [DNN], random forest [RF], and logistic regression [LR]) to explore their ability to predict 1-year mortality by five-fold cross validation methods. We conducted analyses leveraging features from prior to/at and after/at the time of HF diagnosis.Results: The area under the receiver operating curve (AUC) was used to evaluate the performance of the three models: the mean AUC was 0.80 for DNN, 0.72 for RF, and 0.74 for LR. Age, creatinine, body mass index, and blood pressure levels were especially important features in predicting death within 1-year among HF patients.Conclusions: Machine learning models have considerable potential to improve accuracy in mortality prediction, such that high-risk surgical intervention can be applied only in those patients who stand to benefit from it. Access to EHR-based synthetic data derivatives eliminates risk of exposure of EHR data, speeds time-to-insight, and facilitates data sharing. As more clinical, imaging, and contractile features with proven predictive capability are added to these models, the development of a clinical tool to assist in timing of intervention in surgical candidates may be possible.

A group of 6 synthetic datasets is provided with a 914 days water consumption time horizon with structural breaks based on actual datasets from a hotel and hospital. The parameters of the best-fit probability distributions to the actual water consumption data were used to generate these datasets. The distributions used that best fit the datasets used were the gamma distribution for the hotel data set and the gamma and logistics distribution for the hospital dataset. Two structural breaks of 5% and 10% in the mean of the distributions were added to simulate reductions in water consumption patterns.

This model learning dataset is created out of the Raw Synthetic RWD raw dataset, including some of the original attributes. It is distributed in JOBLIB files, where .joblib files contain the vectors and _ids.joblib contain the ID of the person from which each vector is extracted.

This is useful in case it is needed to map the vectors to metadata about the people that are found in the original raw dataset. Note that corresponds to , or , depending on the dataset. The split is roughly 60% of the people are in the training dataset, and 20% in each of the validation and the testing datasets. The input attributes are the age, the short-term averages and the trends of the current week’s BMI, steps walked, calories burned, sleep quality, mood and water consumption, as well as the previous week’s short-term average and trend of the answer to the health self-assessment question.

The outcome to be predicted is the binary quantized health self-assessment answer to be given in the current week. The dataset is normalized based on the training set. The means and standard deviations used can be found in the train_statistics.joblib file. Finally, the output_descriptions.joblib file contains descriptions of the outcomes to be predicted (not actually needed, since included here).

The Synthea Generated Synthetic Data in FHIR hosts over 1 million synthetic patient records generated using Synthea in FHIR format. Exported from the Google Cloud Healthcare API FHIR Store into BigQuery using analytics schema . This public dataset is hosted in Google BigQuery and is included in BigQuery's 1TB/mo of free tier processing. This means that each user receives 1TB of free BigQuery processing every month, which can be used to run queries on this public dataset. Watch this short video to learn how to get started quickly using BigQuery to access public datasets. What is BigQuery . This public dataset is also available in Google Cloud Storage and available free to use. The URL for the GCS bucket is gs://gcp-public-data--synthea-fhir-data-1m-patients. Use this quick start guide to quickly learn how to access public datasets on Google Cloud Storage. Please cite SyntheaTM as: Jason Walonoski, Mark Kramer, Joseph Nichols, Andre Quina, Chris Moesel, Dylan Hall, Carlton Duffett, Kudakwashe Dube, Thomas Gallagher, Scott McLachlan, Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record, Journal of the American Medical Informatics Association, Volume 25, Issue 3, March 2018, Pages 230–238, https://doi.org/10.1093/jamia/ocx079

Context

• We would like to have good open source speech recognition
• Commercial companies try to solve a hard problem: map arbitrary, open-ended speech to text and identify meaning
• The easier problem should be: detect a predefined sequence of sounds and map it to a predefined action.
• Lets tackle the simplest problem first: Classifying single, short words (commands)
• Audio training data is difficult to obtain.

Approaches

• The parent project (spoken verbs) created synthetic speech datasets using text-to-speech programs. The focus there is on single-syllable verbs (commands).
• The Speech Commands dataset (by Pete Warden, see the TensorFlow Speech Recognition Challenge) asked volunteers to pronounce a small set of words: (yes, no, up, down, left, right, on, off, stop, go, and 0-9).
• This data set provides synthetic counterparts to this real world dataset.

Open questions

One can use these two datasets in various ways. Here are some things I am interested in seeing answered:

1. What is it in an audio sample that makes it "sound similar"? Our ears can easily classify both synthetic and real speech, but for algorithms this is still hard. Extending the real dataset with the synthetic data yields a larger training sample and more diversity.
2. How well does an algorithm trained on one data set perform on the other? (transfer learning) If it works poorly, the algorithm probably has not found the key to audio similarity.
3. Are synthetic data sufficient for classifying real datasets? If this is the case, the implications are huge. You would not need to ask thousands of volunteers for hours of time. Instead, you could easily create arbitrary synthetic datasets for your target words.

A interesting challenge (idea for competition) would be to train on this data set and evaluate on the real dataset.

Synthetic data creation

Here I describe how the synthetic audio samples were created. Code is available at https://github.com/JohannesBuchner/spoken-command-recognition, in the "tensorflow-speech-words" folder.

1. The list of words is in "inputwords". "marvin" was changed to "marvel", because "marvin" does not have a pronounciation coding yet.
2. Pronounciations were taken from the British English Example Pronciation dictionary (BEEP, http://svr-www.eng.cam.ac.uk/comp.speech/Section1/Lexical/beep.html ). The phonemes were translated for the next step with a translation table (see compile.py for details). This creates the file "words". There are multiple pronounciations and stresses for each word.
3. A text-to-speech program (espeak) was used to pronounce these words (see generatetfspeech.sh for details). The pronounciation, stress, pitch, speed and speaker were varied. This gives >1000 clean examples for each word.
4. Noise samples were obtained. Noise samples (airport babble car exhibition restaurant street subway train) come from AURORA (https://www.ee.columbia.edu/~dpwe/sounds/noise/), and additional noise samples were synthetically created (ocean white brown pink). (see ../generatenoise.sh for details)
5. Noise and speech were mixed. The speech volume and offset were varied. The noise source, volume was also varied. See addnoise.py for details. addnoise2.py is the same, but with lower speech volume and higher noise volume. All audio files are one second (1s) long and are in wav format (16 bit, mono, 16000 Hz).
6. Finally, the data was compressed into an archive and uploaded to kaggle.

Acknowledgements

This work built upon

• Pronounciation dictionary: BEEP: http://svr-www.eng.cam.ac.uk/comp.speech/Section1/Lexical/beep.html
• Noise samples: AURORA: https://www.ee.columbia.edu/~dpwe/sounds/noise/
• eSPEAK: http://espeak.sourceforge.net/ and mbrola voices http://www.tcts.fpms.ac.be/synthesis/mbrola/mbrcopybin.html

Please provide appropriate citations to the above when using this work.

To cite the resulting dataset, you can use:

APA-style citation: "Buchner J. Synthetic Speech Commands: A public dataset for single-word speech recognition, 2017. Available from https://www.kaggle.com/jbuchner/synthetic-speech-commands-dataset/".

BibTeX @article{speechcommands, title={Synthetic Speech Commands: A public dataset for single-word speech recognition.}, author={Buchner, Johannes}, journal={Dataset available from https://www.kaggle.com/jbuchner/synthetic-speech-commands-dataset/}, year={2017} }

Thanks to everyone trying to improve open source voice detection and speech recognition.

Links

• https://www.kaggle.com/jbuchner/spokenverbs
• https://www.kaggle.com/c/tensorflow-speech-recognition-challenge
• https://github.com/JohannesBuchner/spoken-command-recognition/

This model learning dataset is created out of the Raw Synthetic RWD raw dataset, including some of the original attributes. It is distributed in JOBLIB files, where .joblib files contain the vectors and _ids.joblib contain the ID of the person from which each vector is extracted. This is useful in case it is needed to map the vectors to metadata about the people that are found in the original raw dataset. Note that corresponds to , or , depending on the dataset. The split is roughly 60% of the people are in the training dataset, and 20% in each of the validation and the testing datasets. The input attributes are the age, the short-term averages and the trends of the current week’s BMI, steps walked, calories burned, sleep quality, mood and water consumption, as well as the previous week’s short-term average and trend of the answer to the health self-assessment question. The outcome to be predicted is a tristate quantized version of the health self-assessment answer to be given in the current week. The dataset is normalized based on the training set. The means and standard deviations used can be found in the train_statistics.joblib file. Finally, the output_descriptions.joblib file contains descriptions of the outcomes to be predicted (not actually needed, since included here).