Snapshot img

Synthetic Data Generation Market Size 2025-2029

The synthetic data generation market size is forecast to increase by USD 4.39 billion, at a CAGR of 61.1% between 2024 and 2029.

The market is experiencing significant growth, driven by the escalating demand for data privacy protection. With increasing concerns over data security and the potential risks associated with using real data, synthetic data is gaining traction as a viable alternative. Furthermore, the deployment of large language models is fueling market expansion, as these models can generate vast amounts of realistic and diverse data, reducing the reliance on real-world data sources. However, high costs associated with high-end generative models pose a challenge for market participants. These models require substantial computational resources and expertise to develop and implement effectively. Companies seeking to capitalize on market opportunities must navigate these challenges by investing in research and development to create more cost-effective solutions or partnering with specialists in the field. Overall, the market presents significant potential for innovation and growth, particularly in industries where data privacy is a priority and large language models can be effectively utilized.

What will be the Size of the Synthetic Data Generation Market during the forecast period?

Explore in-depth regional segment analysis with market size data - historical 2019-2023 and forecasts 2025-2029 - in the full report.
Request Free SampleThe market continues to evolve, driven by the increasing demand for data-driven insights across various sectors. Data processing is a crucial aspect of this market, with a focus on ensuring data integrity, privacy, and security. Data privacy-preserving techniques, such as data masking and anonymization, are essential in maintaining confidentiality while enabling data sharing. Real-time data processing and data simulation are key applications of synthetic data, enabling predictive modeling and data consistency. Data management and workflow automation are integral components of synthetic data platforms, with cloud computing and model deployment facilitating scalability and flexibility. Data governance frameworks and compliance regulations play a significant role in ensuring data quality and security. Deep learning models, variational autoencoders (VAEs), and neural networks are essential tools for model training and optimization, while API integration and batch data processing streamline the data pipeline. Machine learning models and data visualization provide valuable insights, while edge computing enables data processing at the source. Data augmentation and data transformation are essential techniques for enhancing the quality and quantity of synthetic data. Data warehousing and data analytics provide a centralized platform for managing and deriving insights from large datasets. Synthetic data generation continues to unfold, with ongoing research and development in areas such as federated learning, homomorphic encryption, statistical modeling, and software development. The market's dynamic nature reflects the evolving needs of businesses and the continuous advancements in data technology.

How is this Synthetic Data Generation Industry segmented?

The synthetic data generation industry research report provides comprehensive data (region-wise segment analysis), with forecasts and estimates in 'USD million' for the period 2025-2029, as well as historical data from 2019-2023 for the following segments. End-userHealthcare and life sciencesRetail and e-commerceTransportation and logisticsIT and telecommunicationBFSI and othersTypeAgent-based modellingDirect modellingApplicationAI and ML Model TrainingData privacySimulation and testingOthersProductTabular dataText dataImage and video dataOthersGeographyNorth AmericaUSCanadaMexicoEuropeFranceGermanyItalyUKAPACChinaIndiaJapanRest of World (ROW)

By End-user Insights

The healthcare and life sciences segment is estimated to witness significant growth during the forecast period.In the rapidly evolving data landscape, the market is gaining significant traction, particularly in the healthcare and life sciences sector. With a growing emphasis on data-driven decision-making and stringent data privacy regulations, synthetic data has emerged as a viable alternative to real data for various applications. This includes data processing, data preprocessing, data cleaning, data labeling, data augmentation, and predictive modeling, among others. Medical imaging data, such as MRI scans and X-rays, are essential for diagnosis and treatment planning. However, sharing real patient data for research purposes or training machine learning algorithms can pose significant privacy risks. Synthetic data generation addresses this challenge by producing realistic medical imaging data, ensuring data privacy while enabling research

The synthetic data generation market is projected to be worth USD 0.3 billion in 2024. The market is anticipated to reach USD 13.0 billion by 2034. The market is further expected to surge at a CAGR of 45.9% during the forecast period 2024 to 2034.

Attributes	Key Insights
Synthetic Data Generation Market Estimated Size in 2024	USD 0.3 billion
Projected Market Value in 2034	USD 13.0 billion
Value-based CAGR from 2024 to 2034	45.9%

Country-wise Insights

Countries	Forecast CAGRs from 2024 to 2034
The United States	46.2%
The United Kingdom	47.2%
China	46.8%
Japan	47.0%
Korea	47.3%

Category-wise Insights

Category	CAGR through 2034
Tabular Data	45.7%
Sandwich Assays	45.5%

Report Scope

Attribute	Details
Estimated Market Size in 2024	US$ 0.3 billion
Projected Market Valuation in 2034	US$ 13.0 billion
Value-based CAGR 2024 to 2034	45.9%
Forecast Period	2024 to 2034
Historical Data Available for	2019 to 2023
Market Analysis	Value in US$ Billion
Key Regions Covered	North America Latin America Western Europe Eastern Europe South Asia and Pacific East Asia The Middle East & Africa
Key Market Segments Covered	Data Type Modeling Type Offering Application End Use Region
Key Countries Profiled	The United States Canada Brazil Mexico Germany France France Spain Italy Russia Poland Czech Republic Romania India Bangladesh Australia New Zealand China Japan South Korea GCC countries South Africa Israel
Key Companies Profiled	Mostly AI CVEDIA Inc. Gretel Labs Datagen NVIDIA Corporation Synthesis AI Amazon.com, Inc. Microsoft Corporation IBM Corporation Meta

Quantum-AI Synthetic Data Generator Market Outlook

According to our latest research, the global Quantum-AI Synthetic Data Generator market size reached USD 1.82 billion in 2024, reflecting a robust expansion driven by technological advancements and increasing adoption across multiple industries. The market is projected to grow at a CAGR of 32.7% from 2025 to 2033, reaching a forecasted market size of USD 21.69 billion by 2033. This growth trajectory is primarily fueled by the rising demand for high-quality synthetic data to train artificial intelligence models, address data privacy concerns, and accelerate digital transformation initiatives across sectors such as healthcare, finance, and retail.

One of the most significant growth factors for the Quantum-AI Synthetic Data Generator market is the escalating need for vast, diverse, and privacy-compliant datasets to train advanced AI and machine learning models. As organizations increasingly recognize the limitations and risks associated with using real-world data, particularly regarding data privacy regulations like GDPR and CCPA, the adoption of synthetic data generation technologies has surged. Quantum computing, when integrated with artificial intelligence, enables the rapid and efficient creation of highly realistic synthetic datasets that closely mimic real-world data distributions while ensuring complete anonymity. This capability is proving invaluable for sectors like healthcare and finance, where data sensitivity is paramount and regulatory compliance is non-negotiable. As a result, organizations are investing heavily in Quantum-AI synthetic data solutions to enhance model accuracy, reduce bias, and streamline data sharing without compromising privacy.

Another key driver propelling the market is the growing complexity and volume of data generated by emerging technologies such as IoT, autonomous vehicles, and smart devices. Traditional data collection methods are often insufficient to keep pace with the data requirements of modern AI applications, leading to gaps in data availability and quality. Quantum-AI Synthetic Data Generators address these challenges by producing large-scale, high-fidelity synthetic datasets on demand, enabling organizations to simulate rare events, test edge cases, and improve model robustness. Additionally, the capability to generate structured, semi-structured, and unstructured data allows businesses to meet the specific needs of diverse applications, ranging from fraud detection in banking to predictive maintenance in manufacturing. This versatility is further accelerating market adoption, as enterprises seek to future-proof their AI initiatives and gain a competitive edge.

The integration of Quantum-AI Synthetic Data Generators into cloud-based platforms and enterprise IT ecosystems is also catalyzing market growth. Cloud deployment models offer scalability, flexibility, and cost-effectiveness, making synthetic data generation accessible to organizations of all sizes, including small and medium enterprises. Furthermore, the proliferation of AI-driven analytics in sectors such as retail, e-commerce, and telecommunications is creating new opportunities for synthetic data applications, from enhancing customer experience to optimizing supply chain operations. As vendors continue to innovate and expand their service offerings, the market is expected to witness sustained growth, with new entrants and established players alike vying for market share through strategic partnerships, product launches, and investments in R&D.

From a regional perspective, North America currently dominates the Quantum-AI Synthetic Data Generator market, accounting for over 38% of the global revenue in 2024, followed by Europe and Asia Pacific. The strong presence of leading technology companies, robust investment in AI research, and favorable regulatory environment contribute to North America's leadership position. Europe is also witnessing significant growth, driven by stringent data privacy regulations and increasing adoption of AI across industries. Meanwhile, the Asia Pacific region is emerging as a high-growth market, fueled by rapid digitalization, expanding IT infrastructure, and government initiatives promoting AI innovation. As regional markets continue to evolve, strategic collaborations and cross-border partnerships are expected to play a pivotal role in shaping the global landscape of the Quantum-AI Synthetic Data Generator market.

Component Analysis

&l

Quantum-AI Synthetic Data Generator Market Outlook

According to our latest research, the global Quantum-AI Synthetic Data Generator market size reached USD 1.98 billion in 2024, reflecting robust momentum driven by the convergence of quantum computing and artificial intelligence technologies in data generation. The market is experiencing a significant compound annual growth rate (CAGR) of 32.1% from 2025 to 2033. At this pace, the market is forecasted to reach USD 24.8 billion by 2033. This remarkable growth is propelled by the escalating demand for high-quality synthetic data across industries to enhance AI model training, ensure data privacy, and overcome data scarcity challenges.

One of the primary growth drivers for the Quantum-AI Synthetic Data Generator market is the increasing reliance on advanced machine learning and deep learning models that require vast amounts of diverse, high-fidelity data. Traditional data sources often fall short in volume, variety, and compliance with privacy regulations. Quantum-AI synthetic data generators address these challenges by producing realistic, representative datasets that mimic real-world scenarios without exposing sensitive information. This capability is particularly crucial in regulated sectors such as healthcare and finance, where data privacy and security are paramount. As organizations seek to accelerate AI adoption while minimizing ethical and legal risks, the demand for sophisticated synthetic data solutions continues to rise.

Another significant factor fueling market expansion is the rapid evolution of quantum computing and its integration with AI algorithms. Quantum computing’s superior processing power enables the generation of complex, large-scale datasets at unprecedented speeds and accuracy. This synergy allows enterprises to simulate intricate data patterns and rare events that would be difficult or impossible to capture through conventional means. Additionally, the proliferation of AI-driven applications in sectors like autonomous vehicles, predictive maintenance, and personalized medicine is amplifying the need for synthetic data generators that can support advanced analytics and model validation. The ongoing advancements in quantum hardware, coupled with the growing ecosystem of AI tools, are expected to further catalyze innovation and adoption in this market.

Moreover, the shift toward digital transformation and the growing adoption of cloud-based solutions are reshaping the landscape of the Quantum-AI Synthetic Data Generator market. Enterprises of all sizes are embracing synthetic data generation to streamline data workflows, reduce operational costs, and accelerate time-to-market for AI-powered products and services. Cloud deployment models offer scalability, flexibility, and seamless integration with existing data infrastructure, making synthetic data generation accessible even to resource-constrained organizations. As digital ecosystems evolve and data-driven decision-making becomes a competitive imperative, the strategic importance of synthetic data generation is set to intensify, fostering sustained market growth through 2033.

From a regional perspective, North America currently leads the market, driven by early technology adoption, substantial investments in quantum and AI research, and a vibrant ecosystem of startups and established technology firms. Europe follows closely, benefiting from strong regulatory frameworks and robust funding for AI innovation. The Asia Pacific region is witnessing the fastest growth, fueled by expanding digital economies, government initiatives supporting AI and quantum technology, and increasing awareness of synthetic data’s strategic value. As global enterprises seek to harness the power of quantum-AI synthetic data generators to gain a competitive edge, regional dynamics will continue to shape market trajectories and opportunities.

Component Analysis

The Component segment of the Quantum-AI Synthetic Data Generator

Synthetic Data Video Generator Market Outlook

According to our latest research, the global synthetic data video generator market size reached USD 1.32 billion in 2024 and is anticipated to grow at a robust CAGR of 38.7% from 2025 to 2033. By the end of 2033, the market is projected to reach USD 18.59 billion, driven by rapid advancements in artificial intelligence, the growing need for high-quality training data for machine learning models, and increasing adoption across industries such as autonomous vehicles, healthcare, and surveillance. The surge in demand for data privacy, coupled with the necessity to overcome data scarcity and bias in real-world datasets, is significantly fueling the synthetic data video generator market's growth trajectory.

One of the primary growth factors for the synthetic data video generator market is the escalating demand for high-fidelity, annotated video datasets required to train and validate AI-driven systems. Traditional data collection methods are often hampered by privacy concerns, high costs, and the sheer complexity of obtaining diverse and representative video samples. Synthetic data video generators address these challenges by enabling the creation of large-scale, customizable, and bias-free datasets that closely mimic real-world scenarios. This capability is particularly vital for sectors such as autonomous vehicles and robotics, where the accuracy and safety of AI models depend heavily on the quality and variety of training data. As organizations strive to accelerate innovation and reduce the risks associated with real-world data collection, the adoption of synthetic data video generation technologies is expected to expand rapidly.

Another significant driver for the synthetic data video generator market is the increasing regulatory scrutiny surrounding data privacy and compliance. With stricter regulations such as GDPR and CCPA coming into force, organizations face mounting challenges in using real-world video data that may contain personally identifiable information. Synthetic data offers an effective solution by generating video datasets devoid of any real individuals, thereby ensuring compliance while still enabling advanced analytics and machine learning. Moreover, synthetic data video generators empower businesses to simulate rare or hazardous events that are difficult or unethical to capture in real life, further enhancing model robustness and preparedness. This advantage is particularly pronounced in healthcare, surveillance, and automotive industries, where data privacy and safety are paramount.

Technological advancements and increasing integration with cloud-based platforms are also propelling the synthetic data video generator market forward. The proliferation of cloud computing has made it easier for organizations of all sizes to access scalable synthetic data generation tools without significant upfront investments in hardware or infrastructure. Furthermore, the continuous evolution of generative adversarial networks (GANs) and other deep learning techniques has dramatically improved the realism and utility of synthetic video data. As a result, companies are now able to generate highly realistic, scenario-specific video datasets at scale, reducing both the time and cost required for AI development. This democratization of synthetic data technology is expected to unlock new opportunities across a wide array of applications, from entertainment content production to advanced surveillance systems.

From a regional perspective, North America currently dominates the synthetic data video generator market, accounting for the largest share in 2024, followed closely by Europe and Asia Pacific. The strong presence of leading AI technology providers, robust investment in research and development, and early adoption by automotive and healthcare sectors are key contributors to North America's market leadership. Europe is also witnessing significant growth, driven by stringent data privacy regulations and increased focus on AI-driven innovation. Meanwhile, Asia Pacific is emerging as a high-growth region, fueled by rapid digital transformation, expanding IT infrastructure, and increasing investments in autonomous systems and smart city projects. Latin America and Middle East & Africa, while still nascent, are expected to experience steady uptake as awareness and technological capabilities continue to grow.

Component Analysis

The synthetic data video generator market by comp

Data Description

We release the synthetic data generated using the method described in the paper Knowledge-Infused Prompting: Assessing and Advancing Clinical Text Data Generation with Large Language Models (ACL 2024 Findings). The external knowledge we use is based on LLM-generated topics and writing styles.

  Generated Datasets

The original train/validation/test data, and the generated synthetic training data are listed as follows. For each dataset, we generate 5000… See the full description on the dataset page: https://huggingface.co/datasets/ritaranx/clinical-synthetic-text-llm.

Synthetic Data Video Generator Market Outlook

According to our latest research, the global Synthetic Data Video Generator market size in 2024 stands at USD 1.46 billion, with robust momentum driven by advances in artificial intelligence and the increasing need for high-quality, privacy-compliant video datasets. The market is witnessing a remarkable compound annual growth rate (CAGR) of 37.2% from 2025 to 2033, propelled by growing adoption across sectors such as autonomous vehicles, healthcare, and surveillance. By 2033, the market is projected to reach USD 18.16 billion, reflecting a seismic shift in how organizations leverage synthetic data to accelerate innovation and mitigate data privacy concerns.

The primary growth factor for the Synthetic Data Video Generator market is the surging demand for data privacy and compliance in machine learning and computer vision applications. As regulatory frameworks like GDPR and CCPA become more stringent, organizations are increasingly wary of using real-world video data that may contain personally identifiable information. Synthetic data video generators provide a scalable and ethical alternative, enabling enterprises to train and validate AI models without risking privacy breaches. This trend is particularly pronounced in sectors such as healthcare and finance, where data sensitivity is paramount. The ability to generate diverse, customizable, and annotation-rich video datasets not only addresses compliance requirements but also accelerates the development and deployment of AI solutions.

Another significant driver is the rapid evolution of deep learning algorithms and simulation technologies, which have dramatically improved the realism and utility of synthetic video data. Innovations in generative adversarial networks (GANs), 3D rendering engines, and advanced simulation platforms have made it possible to create synthetic videos that closely mimic real-world environments and scenarios. This capability is invaluable for industries like autonomous vehicles and robotics, where extensive and varied training data is essential for safe and reliable system behavior. The reduction in time, cost, and logistical complexity associated with collecting and labeling real-world video data further enhances the attractiveness of synthetic data video generators, positioning them as a cornerstone technology for next-generation AI development.

The expanding use cases for synthetic video data across emerging applications also contribute to market growth. Beyond traditional domains such as surveillance and entertainment, synthetic data video generators are finding adoption in areas like augmented reality, smart retail, and advanced robotics. The flexibility to simulate rare, dangerous, or hard-to-capture scenarios offers a strategic advantage for organizations seeking to future-proof their AI initiatives. As synthetic data generation platforms become more accessible and user-friendly, small and medium enterprises are also entering the fray, democratizing access to high-quality training data and fueling a new wave of AI-driven innovation.

From a regional perspective, North America continues to dominate the Synthetic Data Video Generator market, benefiting from a concentration of technology giants, research institutions, and early adopters across key verticals. Europe follows closely, driven by strong regulatory emphasis on data protection and an active ecosystem of AI startups. Meanwhile, the Asia Pacific region is emerging as a high-growth market, buoyed by rapid digital transformation, government AI initiatives, and increasing investments in autonomous systems and smart cities. Latin America and the Middle East & Africa are also showing steady progress, albeit from a smaller base, as awareness and infrastructure for synthetic data generation mature.

Component Analysis

The Synthetic Data Video Generator market, when analyzed by component, is primarily segmented into Software and Services. The software segment currently commands the largest share, driven by the prolif

Ainnotate’s proprietary dataset generation methodology based on large scale generative modelling and Domain randomization provides data that is well balanced with consistent sampling, accommodating rare events, so that it can enable superior simulation and training of your models.

Ainnotate currently provides synthetic datasets in the following domains and use cases.

Internal Services - Visa application, Passport validation, License validation, Birth certificates Financial Services - Bank checks, Bank statements, Pay slips, Invoices, Tax forms, Insurance claims and Mortgage/Loan forms Healthcare - Medical Id cards

The Synthea generated data is provided here as a 1,000 person (1k), 100,000 person (100k), and 2,800,000 persom (2.8m) data sets in the OMOP Common Data Model format. SyntheaTM is a synthetic patient generator that models the medical history of synthetic patients. Our mission is to output high-quality synthetic, realistic but not real, patient data and associated health records covering every aspect of healthcare. The resulting data is free from cost, privacy, and security restrictions. It can be used without restriction for a variety of secondary uses in academia, research, industry, and government (although a citation would be appreciated). You can read our first academic paper here: https://doi.org/10.1093/jamia/ocx079

SDNist (v1.3) is a set of benchmark data and metrics for the evaluation of synthetic data generators on structured tabular data. This version (1.3) reproduces the challenge environment from Sprints 2 and 3 of the Temporal Map Challenge. These benchmarks are distributed as a simple open-source python package to allow standardized and reproducible comparison of synthetic generator models on real world data and use cases. These data and metrics were developed for and vetted through the NIST PSCR Differential Privacy Temporal Map Challenge, where the evaluation tools, k-marginal and Higher Order Conjunction, proved effective in distinguishing competing models in the competition environment.SDNist is available via pip install: pip install sdnist==1.2.8 for Python >=3.6 or on the USNIST/Github. The sdnist Python module will download data from NIST as necessary, and users are not required to download data manually.

With the ongoing energy transition, power grids are evolving fast. They operate more and more often close to their technical limit, under more and more volatile conditions. Fast, essentially real-time computational approaches to evaluate their operational safety, stability and reliability are therefore highly desirable. Machine Learning methods have been advocated to solve this challenge, however they are heavy consumers of training and testing data, while historical operational data for real-world power grids are hard if not impossible to access.

This dataset contains long time series for production, consumption, and line flows, amounting to 20 years of data with a time resolution of one hour, for several thousands of loads and several hundreds of generators of various types representing the ultra-high-voltage transmission grid of continental Europe. The synthetic time series have been statistically validated agains real-world data.

Data generation algorithm

The algorithm is described in a Nature Scientific Data paper. It relies on the PanTaGruEl model of the European transmission network -- the admittance of its lines as well as the location, type and capacity of its power generators -- and aggregated data gathered from the ENTSO-E transparency platform, such as power consumption aggregated at the national level.

Network

The network information is encoded in the file europe_network.json. It is given in PowerModels format, which it itself derived from MatPower and compatible with PandaPower. The network features 7822 power lines and 553 transformers connecting 4097 buses, to which are attached 815 generators of various types.

Time series

The time series forming the core of this dataset are given in CSV format. Each CSV file is a table with 8736 rows, one for each hourly time step of a 364-day year. All years are truncated to exactly 52 weeks of 7 days, and start on a Monday (the load profiles are typically different during weekdays and weekends). The number of columns depends on the type of table: there are 4097 columns in load files, 815 for generators, and 8375 for lines (including transformers). Each column is described by a header corresponding to the element identifier in the network file. All values are given in per-unit, both in the model file and in the tables, i.e. they are multiples of a base unit taken to be 100 MW.

There are 20 tables of each type, labeled with a reference year (2016 to 2020) and an index (1 to 4), zipped into archive files arranged by year. This amount to a total of 20 years of synthetic data. When using loads, generators, and lines profiles together, it is important to use the same label: for instance, the files loads_2020_1.csv, gens_2020_1.csv, and lines_2020_1.csv represent a same year of the dataset, whereas gens_2020_2.csv is unrelated (it actually shares some features, such as nuclear profiles, but it is based on a dispatch with distinct loads).

Usage

The time series can be used without a reference to the network file, simply using all or a selection of columns of the CSV files, depending on the needs. We show below how to select series from a particular country, or how to aggregate hourly time steps into days or weeks. These examples use Python and the data analyis library pandas, but other frameworks can be used as well (Matlab, Julia). Since all the yearly time series are periodic, it is always possible to define a coherent time window modulo the length of the series.

Selecting a particular country

This example illustrates how to select generation data for Switzerland in Python. This can be done without parsing the network file, but using instead gens_by_country.csv, which contains a list of all generators for any country in the network. We start by importing the pandas library, and read the column of the file corresponding to Switzerland (country code CH):

import pandas as pd
CH_gens = pd.read_csv('gens_by_country.csv', usecols=['CH'], dtype=str)

The object created in this way is Dataframe with some null values (not all countries have the same number of generators). It can be turned into a list with:

CH_gens_list = CH_gens.dropna().squeeze().to_list()

Finally, we can import all the time series of Swiss generators from a given data table with

pd.read_csv('gens_2016_1.csv', usecols=CH_gens_list)

The same procedure can be applied to loads using the list contained in the file loads_by_country.csv.

Averaging over time

This second example shows how to change the time resolution of the series. Suppose that we are interested in all the loads from a given table, which are given by default with a one-hour resolution:

hourly_loads = pd.read_csv('loads_2018_3.csv')

To get a daily average of the loads, we can use:

daily_loads = hourly_loads.groupby([t // 24 for t in range(24 * 364)]).mean()

This results in series of length 364. To average further over entire weeks and get series of length 52, we use:

weekly_loads = hourly_loads.groupby([t // (24 * 7) for t in range(24 * 364)]).mean()

Source code

The code used to generate the dataset is freely available at https://github.com/GeeeHesso/PowerData. It consists in two packages and several documentation notebooks. The first package, written in Python, provides functions to handle the data and to generate synthetic series based on historical data. The second package, written in Julia, is used to perform the optimal power flow. The documentation in the form of Jupyter notebooks contains numerous examples on how to use both packages. The entire workflow used to create this dataset is also provided, starting from raw ENTSO-E data files and ending with the synthetic dataset given in the repository.

Funding

This work was supported by the Cyber-Defence Campus of armasuisse and by an internal research grant of the Engineering and Architecture domain of HES-SO.

We also present the synthesizers that best preserve PPV and TPR accross subgroups. We present the two best synthetic data generator for each task. We selected best synthesizer and runner up based on experiments with privacy-loss budget ϵ = 5.0.

Data model to generate datasets used in the tests of the article: Synthetic Datasets Generator for Testing Techniques and Tools of Information Visualization and Machine Learning.

Dataset Card for 🥤SODA Synthetic Dialogue

  Dataset Summary

🥤SODA Synthetic Dialogue is a set of synthetic dialogues between Assistant and User. In each conversation, User asks Assistant to perform summarization or story generation tasks based on a snippet of an existing dialogue, story, or from a title or theme. This data was created by synthesizing the dialogues in 🥤Soda and applying a set of templates to generate the conversation. The original research paper can be… See the full description on the dataset page: https://huggingface.co/datasets/emozilla/soda_synthetic_dialogue.

Multiturn Multimodal

We want to generate synthetic data that able to understand position and relationship between multi-images and multi-audio, example as below, All notebooks at https://github.com/mesolitica/malaysian-dataset/tree/master/chatbot/multiturn-multimodal

  multi-images

synthetic-multi-images-relationship.jsonl, 100000 rows, 109MB. Images at https://huggingface.co/datasets/mesolitica/translated-LLaVA-Pretrain/tree/main

  Example data

{'filename':… See the full description on the dataset page: https://huggingface.co/datasets/mesolitica/synthetic-multiturn-multimodal.

Artifacts for the paper titled Root Cause Analysis for Microservice System based on Causal Inference: How Far Are We?.

This artifact repository contains 9 compressed folders, as follows:

ID File Name Description

1 syn_circa.zip CIRCA10, and CIRCA50 datasets for Causal Discovery

2 syn_rcd.zip RCD10, and RCD50 datasets for Causal Discovery

3 syn_causil.zip CausIL10, and CausIL50 datasets for Causal Discovery

4 rca_circa.zip CIRCA10, and CIRCA50 datasets for RCA

5 rca_rcd.zip RCD10, and RCD50 datasets for RCA

6 online-boutique.zip Online Boutique dataset for RCA

7 sock-shop-1.zip Sock Shop 1 dataset for RCA

8 sock-shop-2.zip Sock Shop 2 dataset for RCA

9 train-ticket.zip Train Ticket dataset for RCA

Each zip file contains the generated/collected data from the corresponding data generator or microservice benchmark systems (e.g., online-boutique.zip contains metrics data collected from the Online Boutique system).

Details about the generation of our datasets

1. Synthetic datasets

We use three different synthetic data generators from three previous RCA studies [15, 25, 28] to create the synthetic datasets: CIRCA, RCD, and CausIL data generators. Their mechanisms are as follows:1. CIRCA datagenerator [28] generates a random causal directed acyclic graph (DAG) based on a given number of nodes and edges. From this DAG, time series data for each node is generated using a vector auto-regression (VAR) model. A fault is injected into a node by altering the noise term in the VAR model for two timestamps. 2. RCD data generator [25] uses the pyAgrum package [3] to generate a random DAG based on a given number of nodes, subsequently generating discrete time series data for each node, with values ranging from 0 to 5. A fault is introduced into a node by changing its conditional probability distribution.3. CausIL data generator [15] generates causal graphs and time series data that simulate the behavior of microservice systems. It first constructs a DAG of services and metrics based on domain knowledge, then generates metric data for each node of the DAG using regressors trained on real metrics data. Unlike the CIRCA and RCD data generators, the CausIL data generator does not have the capability to inject faults.To create our synthetic datasets, we first generate 10 DAGs whose nodes range from 10 to 50 for each of the synthetic data generators. Next, we generate fault-free datasets using these DAGs with different seedings, resulting in 100 cases for the CIRCA and RCD generators and 10 cases for the CausIL generator. We then create faulty datasets by introducing ten faults into each DAG and generating the corresponding faulty data, yielding 100 cases for the CIRCA and RCD data generators. The fault-free datasets (e.g. syn_rcd, syn_circa) are used to evaluate causal discovery methods, while the faulty datasets (e.g. rca_rcd, rca_circa) are used to assess RCA methods.

1. Data collected from benchmark microservice systems

We deploy three popular benchmark microservice systems: Sock Shop [6], Online Boutique [4], and Train Ticket [8], on a four-node Kubernetes cluster hosted by AWS. Next, we use the Istio service mesh [2] with Prometheus [5] and cAdvisor [1] to monitor and collect resource-level and service-level metrics of all services, as in previous works [ 25 , 39, 59 ]. To generate traffic, we use the load generators provided by these systems and customise them to explore all services with 100 to 200 users concurrently. We then introduce five common faults (CPU hog, memory leak, disk IO stress, network delay, and packet loss) into five different services within each system. Finally, we collect metrics data before and after the fault injection operation. An overview of our setup is presented in the Figure below.

Code

The code to reproduce the experimental results in the paper is available at https://github.com/phamquiluan/RCAEval.

References

As in our paper.

SPIDER - Synthetic Person Information Dataset for Entity Resolution offers researchers with ready to use data that can be utilized in benchmarking Duplicate or Entity Resolution algorithms. The dataset is aimed at person-level fields that are typical in customer data. As it is hard to source real world person level data due to Personally Identifiable Information (PII), there are very few synthetic data available publicly. The current datasets also come with limitations of small volume and core person-level fields missing in the dataset. SPIDER addresses the challenges by focusing on core person level attributes - first/last name, email, phone, address and dob. Using Python Faker library, 40,000 unique, synthetic person records are created. An additional 10,000 duplicate records are generated from the base records using 7 real-world transformation rules. The duplicate records are labelled with original base record and the duplicate rule used for record generation through is_duplicate_of and duplication_rule fieldsDuplicate RulesDuplicate record with a variation in email address.Duplicate record with a variation in email addressDuplicate record with last name variationDuplicate record with first name variationDuplicate record with a nicknameDuplicate record with near exact spellingDuplicate record with only same email and nameOutput FormatThe dataset is presented in both JSON and CSV formats for use in data processing and machine learning tools.Data RegenerationThe project includes the python script used for generating the 50,000 person records. The Python script can be expanded to include - additional duplicate rules, fuzzy name, geographical names' variations and volume adjustments.Files Includedspider_dataset_20250714_035016.csvspider_dataset_20250714_035016.jsonspider_readme.mdDataDescriptionspythoncodeV1.py

The Insider Threat Test Dataset is a collection of synthetic insider threat test datasets that provide both background and malicious actor synthetic data.The CERT Division, in partnership with ExactData, LLC, and under sponsorship from DARPA I2O, generated a collection of synthetic insider threat test datasets. These datasets provide both synthetic background data and data from synthetic malicious actors.For more background on this data, please see the paper, Bridging the Gap: A Pragmatic Approach to Generating Insider Threat Data.Datasets are organized according to the data generator release that created them. Most releases include multiple datasets (e.g., r3.1 and r3.2). Generally, later releases include a superset of the data generation functionality of earlier releases. Each dataset file contains a readme file that provides detailed notes about the features of that release.The answer key file answers.tar.bz2 contains the details of the malicious activity included in each dataset, including descriptions of the scenarios enacted and the identifiers of the synthetic users involved.

Data generation in machine learning involves creating or manipulating data to train and evaluate machine learning models. The purpose of data generation is to provide diverse and representative examples that cover a wide range of scenarios, ensuring the model's robustness and generalization. Data augmentation techniques involve applying various transformations to existing data samples to create new ones. These transformations include: random rotations, translations, scaling, flips, and more. Augmentation helps in increasing the dataset size, introducing natural variations, and improving model performance by making it more invariant to specific transformations. The dataset contains GENERATED USA passports, which are replicas of official passports but with randomly generated details, such as name, date of birth etc. The primary intention of generating these fake passports is to demonstrate the structure and content of a typical passport document and to train the neural network to identify this type of document. Generated passports can assist in conducting research without accessing or compromising real user data that is often sensitive and subject to privacy regulations. Synthetic data generation allows researchers to develop and refine models using simulated passport data without risking privacy leaks.

Code and experiment results for a synthetic knowledge graph generator. The generator receives a set of rules, with an expected body support and support, and returns a knowledge graph that approximately matches the rules according to the body support and confidence. This code was developed during the Bachelor thesis by Gabriel Glaser, Generating Random Knowledge Graphs from Rules, University of Stuttgart, 2024. doi:10.18419/opus-15467.