Test Data Generation Tools Market Outlook

The global market size for Test Data Generation Tools was valued at USD 800 million in 2023 and is projected to reach USD 2.2 billion by 2032, growing at a CAGR of 12.1% during the forecast period. The surge in the adoption of agile and DevOps practices, along with the increasing complexity of software applications, is driving the growth of this market.

One of the primary growth factors for the Test Data Generation Tools market is the increasing need for high-quality test data in software development. As businesses shift towards more agile and DevOps methodologies, the demand for automated and efficient test data generation solutions has surged. These tools help in reducing the time required for test data creation, thereby accelerating the overall software development lifecycle. Additionally, the rise in digital transformation across various industries has necessitated the need for robust testing frameworks, further propelling the market growth.

The proliferation of big data and the growing emphasis on data privacy and security are also significant contributors to market expansion. With the introduction of stringent regulations like GDPR and CCPA, organizations are compelled to ensure that their test data is compliant with these laws. Test Data Generation Tools that offer features like data masking and data subsetting are increasingly being adopted to address these compliance requirements. Furthermore, the increasing instances of data breaches have underscored the importance of using synthetic data for testing purposes, thereby driving the demand for these tools.

Another critical growth factor is the technological advancements in artificial intelligence and machine learning. These technologies have revolutionized the field of test data generation by enabling the creation of more realistic and comprehensive test data sets. Machine learning algorithms can analyze large datasets to generate synthetic data that closely mimics real-world data, thus enhancing the effectiveness of software testing. This aspect has made AI and ML-powered test data generation tools highly sought after in the market.

Regional outlook for the Test Data Generation Tools market shows promising growth across various regions. North America is expected to hold the largest market share due to the early adoption of advanced technologies and the presence of major software companies. Europe is also anticipated to witness significant growth owing to strict regulatory requirements and increased focus on data security. The Asia Pacific region is projected to grow at the highest CAGR, driven by rapid industrialization and the growing IT sector in countries like India and China.

Synthetic Data Generation has emerged as a pivotal component in the realm of test data generation tools. This process involves creating artificial data that closely resembles real-world data, without compromising on privacy or security. The ability to generate synthetic data is particularly beneficial in scenarios where access to real data is restricted due to privacy concerns or regulatory constraints. By leveraging synthetic data, organizations can perform comprehensive testing without the risk of exposing sensitive information. This not only ensures compliance with data protection regulations but also enhances the overall quality and reliability of software applications. As the demand for privacy-compliant testing solutions grows, synthetic data generation is becoming an indispensable tool in the software development lifecycle.

Component Analysis

The Test Data Generation Tools market is segmented into software and services. The software segment is expected to dominate the market throughout the forecast period. This dominance can be attributed to the increasing adoption of automated testing tools and the growing need for robust test data management solutions. Software tools offer a wide range of functionalities, including data profiling, data masking, and data subsetting, which are essential for effective software testing. The continuous advancements in software capabilities also contribute to the growth of this segment.

In contrast, the services segment, although smaller in market share, is expected to grow at a substantial rate. Services include consulting, implementation, and support services, which are crucial for the successful deployment and management of test data generation tools. The increasing complexity of IT inf

Dataset used in the article entitled 'Synthetic Datasets Generator for Testing Information Visualization and Machine Learning Techniques and Tools'. These datasets can be used to test several characteristics in machine learning and data processing algorithms.

The Test Data Generation Tools market is experiencing robust growth, driven by the increasing demand for high-quality software and the rising adoption of agile and DevOps methodologies. The market's expansion is fueled by several factors, including the need for realistic and representative test data to ensure thorough software testing, the growing complexity of applications, and the increasing pressure to accelerate software delivery cycles. The market is segmented by type (Random, Pathwise, Goal, Intelligent) and application (Large Enterprises, SMEs), each demonstrating unique growth trajectories. Intelligent test data generation, offering advanced capabilities like data masking and synthetic data creation, is gaining significant traction, while large enterprises are leading the adoption due to their higher testing volumes and budgets. Geographically, North America and Europe currently hold the largest market shares, but the Asia-Pacific region is expected to witness significant growth due to rapid digitalization and increasing software development activities. Competitive intensity is high, with a mix of established players like IBM and Informatica and emerging innovative companies continuously introducing advanced features and functionalities. The market's growth is, however, constrained by challenges such as the complexity of implementing and managing test data generation tools and the need for specialized expertise. Overall, the market is projected to maintain a healthy growth rate throughout the forecast period (2025-2033), driven by continuous technological advancements and evolving software testing requirements. While the precise CAGR isn't provided, assuming a conservative yet realistic CAGR of 15% based on industry trends and the factors mentioned above, the market is poised for significant expansion. This growth will be fueled by the increasing adoption of cloud-based solutions, improved data masking techniques for enhanced security and privacy, and the rise of AI-powered test data generation tools that automatically create comprehensive and realistic datasets. The competitive landscape will continue to evolve, with mergers and acquisitions likely shaping the market structure. Furthermore, the focus on data privacy regulations will influence the development and adoption of advanced data anonymization and synthetic data generation techniques. The market will see further segmentation as specialized tools catering to specific industry needs (e.g., financial services, healthcare) emerge. The long-term outlook for the Test Data Generation Tools market remains positive, driven by the relentless demand for higher software quality and faster development cycles.

The Test Data Generation Tools market is experiencing robust growth, driven by the increasing demand for efficient and reliable software testing in a rapidly evolving digital landscape. The market's expansion is fueled by several key factors: the escalating complexity of software applications, the growing adoption of agile and DevOps methodologies which necessitate faster test cycles, and the rising need for high-quality software releases to meet stringent customer expectations. Organizations across various sectors, including finance, healthcare, and technology, are increasingly adopting test data generation tools to automate the creation of realistic and representative test data, thereby reducing testing time and costs while enhancing the overall quality of software products. This shift is particularly evident in the adoption of cloud-based solutions, offering scalability and accessibility benefits. The competitive landscape is marked by a mix of established players like IBM and Microsoft, alongside specialized vendors like Broadcom and Informatica, and emerging innovative startups. The market is witnessing increased mergers and acquisitions as larger players seek to expand their market share and product portfolios. Future growth will be influenced by advancements in artificial intelligence (AI) and machine learning (ML), enabling the generation of even more realistic and sophisticated test data, further accelerating market expansion. The market's projected Compound Annual Growth Rate (CAGR) suggests a substantial increase in market value over the forecast period (2025-2033). While precise figures were not provided, a reasonable estimation based on current market trends indicates a significant expansion. Market segmentation will likely see continued growth across various sectors, with cloud-based solutions gaining traction. Geographic expansion will also contribute to overall growth, particularly in regions with rapidly developing software industries. However, challenges remain, such as the need for skilled professionals to manage and utilize these tools effectively and the potential security concerns related to managing large datasets. Addressing these challenges will be crucial for sustained market growth and wider adoption. The overall outlook for the Test Data Generation Tools market remains positive, driven by the persistent need for efficient and robust software testing processes in a continuously evolving technological environment.

The Test Data Generation Tools market is rapidly evolving, driven by the increasing need for high-quality software and data integrity across various industries. Test data generation tools are essential in the software development lifecycle, enabling organizations to create realistic, secure, and compliant datasets f

Testing web APIs automatically requires generating input data values such as addressess, coordinates or country codes. Generating meaningful values for these types of parameters randomly is rarely feasible, which means a major obstacle for current test case generation approaches. In this paper, we present ARTE, the first semantic-based approach for the Automated generation of Realistic TEst inputs for web APIs. Specifically, ARTE leverages the specification of the API under test to extract semantically related values for every parameter by applying knowledge extraction techniques. Our approach has been integrated into RESTest, a state-of-the-art tool for API testing, achieving an unprecedented level of automation which allows to generate up to 100\% more valid API calls than existing fuzzing techniques (30\% on average). Evaluation results on a set of 26 real-world APIs show that ARTE can generate realistic inputs for 7 out of every 10 parameters, outperforming the results obtained by related approaches.

Introduction

This repository hosts the Testing Roads for Autonomous VEhicLes (TRAVEL) dataset. TRAVEL is an extensive collection of virtual roads that have been used for testing lane assist/keeping systems (i.e., driving agents) and data from their execution in state of the art, physically accurate driving simulator, called BeamNG.tech. Virtual roads consist of sequences of road points interpolated using Cubic splines.

Along with the data, this repository contains instructions on how to install the tooling necessary to generate new data (i.e., test cases) and analyze them in the context of test regression. We focus on test selection and test prioritization, given their importance for developing high-quality software following the DevOps paradigms.

This dataset builds on top of our previous work in this area, including work on

test generation (e.g., AsFault, DeepJanus, and DeepHyperion) and the SBST CPS tool competition (SBST2021),

test selection: SDC-Scissor and related tool

test prioritization: automated test cases prioritization work for SDCs.

Dataset Overview

The TRAVEL dataset is available under the data folder and is organized as a set of experiments folders. Each of these folders is generated by running the test-generator (see below) and contains the configuration used for generating the data (experiment_description.csv), various statistics on generated tests (generation_stats.csv) and found faults (oob_stats.csv). Additionally, the folders contain the raw test cases generated and executed during each experiment (test..json).

The following sections describe what each of those files contains.

Experiment Description

The experiment_description.csv contains the settings used to generate the data, including:

Time budget. The overall generation budget in hours. This budget includes both the time to generate and execute the tests as driving simulations.

The size of the map. The size of the squared map defines the boundaries inside which the virtual roads develop in meters.

The test subject. The driving agent that implements the lane-keeping system under test. The TRAVEL dataset contains data generated testing the BeamNG.AI and the end-to-end Dave2 systems.

The test generator. The algorithm that generated the test cases. The TRAVEL dataset contains data obtained using various algorithms, ranging from naive and advanced random generators to complex evolutionary algorithms, for generating tests.

The speed limit. The maximum speed at which the driving agent under test can travel.

Out of Bound (OOB) tolerance. The test cases' oracle that defines the tolerable amount of the ego-car that can lie outside the lane boundaries. This parameter ranges between 0.0 and 1.0. In the former case, a test failure triggers as soon as any part of the ego-vehicle goes out of the lane boundary; in the latter case, a test failure triggers only if the entire body of the ego-car falls outside the lane.

Experiment Statistics

The generation_stats.csv contains statistics about the test generation, including:

Total number of generated tests. The number of tests generated during an experiment. This number is broken down into the number of valid tests and invalid tests. Valid tests contain virtual roads that do not self-intersect and contain turns that are not too sharp.

Test outcome. The test outcome contains the number of passed tests, failed tests, and test in error. Passed and failed tests are defined by the OOB Tolerance and an additional (implicit) oracle that checks whether the ego-car is moving or standing. Tests that did not pass because of other errors (e.g., the simulator crashed) are reported in a separated category.

The TRAVEL dataset also contains statistics about the failed tests, including the overall number of failed tests (total oob) and its breakdown into OOB that happened while driving left or right. Further statistics about the diversity (i.e., sparseness) of the failures are also reported.

Test Cases and Executions

Each test..json contains information about a test case and, if the test case is valid, the data observed during its execution as driving simulation.

The data about the test case definition include:

The road points. The list of points in a 2D space that identifies the center of the virtual road, and their interpolation using cubic splines (interpolated_points)

The test ID. The unique identifier of the test in the experiment.

Validity flag and explanation. A flag that indicates whether the test is valid or not, and a brief message describing why the test is not considered valid (e.g., the road contains sharp turns or the road self intersects)

The test data are organized according to the following JSON Schema and can be interpreted as RoadTest objects provided by the tests_generation.py module.

{ "type": "object", "properties": { "id": { "type": "integer" }, "is_valid": { "type": "boolean" }, "validation_message": { "type": "string" }, "road_points": { §\label{line:road-points}§ "type": "array", "items": { "$ref": "schemas/pair" }, }, "interpolated_points": { §\label{line:interpolated-points}§ "type": "array", "items": { "$ref": "schemas/pair" }, }, "test_outcome": { "type": "string" }, §\label{line:test-outcome}§ "description": { "type": "string" }, "execution_data": { "type": "array", "items": { "$ref" : "schemas/simulationdata" } } }, "required": [ "id", "is_valid", "validation_message", "road_points", "interpolated_points" ] }

Finally, the execution data contain a list of timestamped state information recorded by the driving simulation. State information is collected at constant frequency and includes absolute position, rotation, and velocity of the ego-car, its speed in Km/h, and control inputs from the driving agent (steering, throttle, and braking). Additionally, execution data contain OOB-related data, such as the lateral distance between the car and the lane center and the OOB percentage (i.e., how much the car is outside the lane).

The simulation data adhere to the following (simplified) JSON Schema and can be interpreted as Python objects using the simulation_data.py module.

{ "$id": "schemas/simulationdata", "type": "object", "properties": { "timer" : { "type": "number" }, "pos" : { "type": "array", "items":{ "$ref" : "schemas/triple" } } "vel" : { "type": "array", "items":{ "$ref" : "schemas/triple" } } "vel_kmh" : { "type": "number" }, "steering" : { "type": "number" }, "brake" : { "type": "number" }, "throttle" : { "type": "number" }, "is_oob" : { "type": "number" }, "oob_percentage" : { "type": "number" } §\label{line:oob-percentage}§ }, "required": [ "timer", "pos", "vel", "vel_kmh", "steering", "brake", "throttle", "is_oob", "oob_percentage" ] }

Dataset Content

The TRAVEL dataset is a lively initiative so the content of the dataset is subject to change. Currently, the dataset contains the data collected during the SBST CPS tool competition, and data collected in the context of our recent work on test selection (SDC-Scissor work and tool) and test prioritization (automated test cases prioritization work for SDCs).

SBST CPS Tool Competition Data

The data collected during the SBST CPS tool competition are stored inside data/competition.tar.gz. The file contains the test cases generated by Deeper, Frenetic, AdaFrenetic, and Swat, the open-source test generators submitted to the competition and executed against BeamNG.AI with an aggression factor of 0.7 (i.e., conservative driver).

    Name
    Map Size (m x m)
    Max Speed (Km/h)
    Budget (h)
    OOB Tolerance (%)
    Test Subject




    DEFAULT
    200 × 200
    120
    5 (real time)
    0.95
    BeamNG.AI - 0.7


    SBST
    200 × 200
    70
    2 (real time)
    0.5
    BeamNG.AI - 0.7

Specifically, the TRAVEL dataset contains 8 repetitions for each of the above configurations for each test generator totaling 64 experiments.

SDC Scissor

With SDC-Scissor we collected data based on the Frenetic test generator. The data is stored inside data/sdc-scissor.tar.gz. The following table summarizes the used parameters.

    Name
    Map Size (m x m)
    Max Speed (Km/h)
    Budget (h)
    OOB Tolerance (%)
    Test Subject




    SDC-SCISSOR
    200 × 200
    120
    16 (real time)
    0.5
    BeamNG.AI - 1.5

The dataset contains 9 experiments with the above configuration. For generating your own data with SDC-Scissor follow the instructions in its repository.

Dataset Statistics

Here is an overview of the TRAVEL dataset: generated tests, executed tests, and faults found by all the test generators grouped by experiment configuration. Some 25,845 test cases are generated by running 4 test generators 8 times in 2 configurations using the SBST CPS Tool Competition code pipeline (SBST in the table). We ran the test generators for 5 hours, allowing the ego-car a generous speed limit (120 Km/h) and defining a high OOB tolerance (i.e., 0.95), and we also ran the test generators using a smaller generation budget (i.e., 2 hours) and speed limit (i.e., 70 Km/h) while setting the OOB tolerance to a lower value (i.e., 0.85). We also collected some 5, 971 additional tests with SDC-Scissor (SDC-Scissor in the table) by running it 9 times for 16 hours using Frenetic as a test generator and defining a more realistic OOB tolerance (i.e., 0.50).

Generating new Data

Generating new data, i.e., test cases, can be done using the SBST CPS Tool Competition pipeline and the driving simulator BeamNG.tech.

Extensive instructions on how to install both software are reported inside the SBST CPS Tool Competition pipeline Documentation;

The appendix of our ICSE 2018 paper "Search-Based Test Data Generation for SQL Queries: Appendix".

The appendix contains:

The queries from the three open source systems we used in the evaluation of our tool (the industry software system is not part of this appendix, due to privacy reasons)

The results of our evaluation.

The source code of the tool. Most recent version can be found at https://github.com/SERG-Delft/evosql.

The results of the tuning procedure we conducted before running the final evaluation.

The global database testing tool market is anticipated to experience substantial growth in the coming years, driven by factors such as the increasing adoption of cloud-based technologies, the rising demand for data quality and accuracy, and the growing complexity of database systems. The market is expected to reach a value of USD 1,542.4 million by 2033, expanding at a CAGR of 7.5% during the forecast period of 2023-2033. Key players in the market include Apache JMeter, DbFit, SQLMap, Mockup Data, SQL Test, NoSQLUnit, Orion, ApexSQL, QuerySurge, DBUnit, DataFactory, DTM Data Generator, Oracle, SeLite, SLOB, and others. The North American region is anticipated to hold a significant share of the database testing tool market, followed by Europe and Asia Pacific. The increasing adoption of cloud-based database testing services, the presence of key market players, and the growing demand for data testing and validation are driving the market growth in North America. Asia Pacific, on the other hand, is expected to experience the highest growth rate due to the rapidly increasing IT spending, the emergence of new technologies, and the growing number of businesses investing in data quality management solutions.

Test Data Management Market size was valued at USD 1.54 Billion in 2024 and is projected to reach USD 2.97 Billion by 2032, growing at a CAGR of 11.19% from 2026 to 2032.

Test Data Management Market Drivers

Increasing Data Volumes: The exponential growth in data generated by businesses necessitates efficient management of test data. Effective TDM solutions help organizations handle large volumes of data, ensuring accurate and reliable testing processes.

Need for Regulatory Compliance: Stringent data privacy regulations, such as GDPR, HIPAA, and CCPA, require organizations to protect sensitive data. TDM solutions help ensure compliance by masking or anonymizing sensitive data used in testing environments.

Published at the 46th International Conference on Software Engineering (ICSE 2024). Here you can find a preprint.About the artifactsdataset.csv.gzeach row represents one test casecolumn "test_type": was the generated or developer-writtencolumn "flaky": has the test shown flaky behavior, and what kind? (NOD = non-order-dependent, OD = order-dependent)used to answer RQ1 (Prevalence) and RQ2 (Flakiness Suppression).LoC.zipcontains lines-of-code data for the Java and Python projectsflaky_java_projects.zip and flaky_python_projects.ziparchives containing the 418 Java and 531 Python projects that contained at least one flaky testeach project contains the developer written and generated test suitesmanual_rootCausing.zipresults of the manual root cause classificationfull_sample.csvcolumn "rater": which of the four researchers conducting the classification rated this test (alignment = all four)used to answer RQ3 (Root Causes)Running the jupyter notebookDownload all artifactsCreate and activate virtual environmentvirtualenv -p venvsource venv/bin/activateInstall dependenciespip install -r requirements.txtStart jupyter labpython -m jupyter labScripts used for test generation and executionJava (EvoSuite)Python (Pynguin)

Database Testing Tool Market Outlook

The global database testing tool market size was valued at approximately USD 3.2 billion in 2023 and is expected to reach USD 7.8 billion by 2032, growing at a CAGR of 10.5% during the forecast period. Factors such as the increasing volume of data generated by organizations and the need for robust data management solutions are driving the market growth.

One of the primary growth factors for the database testing tool market is the exponential increase in data generation across various industries. The advent of big data, IoT, and other data-intensive technologies has resulted in massive amounts of data being generated daily. This surge in data necessitates the need for efficient testing tools to ensure data accuracy, integrity, and security, which in turn drives the demand for database testing tools. Moreover, as businesses increasingly rely on data-driven decision-making, the importance of maintaining high data quality becomes paramount, further propelling market growth.

Another significant factor contributing to the growth of this market is the increasing adoption of cloud computing and cloud-based services. Cloud platforms offer scalable and flexible solutions for data storage and management, making it easier for companies to handle large volumes of data. As more organizations migrate to the cloud, the need for effective database testing tools that can operate seamlessly in cloud environments becomes critical. This trend is expected to drive market growth as cloud adoption continues to rise across various industries.

In the realm of software development, the use of Software Testing Tools is becoming increasingly critical. These tools are designed to automate the testing process, ensuring that software applications function correctly and meet specified requirements. By employing Software Testing Tools, organizations can significantly reduce the time and effort required for manual testing, allowing their teams to focus on more strategic tasks. Furthermore, these tools help in identifying bugs and issues early in the development cycle, thereby reducing the cost and time associated with fixing defects later. As the complexity of software applications continues to grow, the demand for advanced Software Testing Tools is expected to rise, driving innovation and development in this sector.

Additionally, regulatory compliance and data governance requirements are playing a crucial role in the growth of the database testing tool market. Governments and regulatory bodies across the globe have implemented stringent data protection and privacy laws, compelling organizations to ensure that their data management practices adhere to these regulations. Database testing tools help organizations meet compliance requirements by validating data integrity, security, and performance, thereby mitigating the risk of non-compliance and associated penalties. This regulatory landscape is expected to further boost the demand for database testing tools.

On the regional front, North America is anticipated to hold a significant share of the database testing tool market due to the presence of major technology companies and a robust IT infrastructure. The region's early adoption of advanced technologies and a strong focus on data management solutions contribute to its market dominance. Europe is also expected to witness substantial growth, driven by stringent data protection regulations such as GDPR and the increasing adoption of cloud services. The Asia Pacific region is projected to exhibit the highest growth rate during the forecast period, owing to the rapid digital transformation, rising adoption of cloud computing, and growing awareness of data quality and security among enterprises.

Type Analysis

The database testing tool market is segmented by type into manual testing tools and automated testing tools. Manual testing tools involve human intervention to execute test cases and analyze results, making them suitable for small-scale applications or projects with limited complexity. However, the manual testing approach can be time-consuming and prone to human errors, which can affect the accuracy and reliability of the test results. Despite these limitations, manual testing tools are still favored in scenarios where precise control and detailed observations are required.

Automated testing tools, on the other hand, have gained significant traction due to their ability to execute a large

Introduction

This datasets have SQL injection attacks (SLQIA) as malicious Netflow data. The attacks carried out are SQL injection for Union Query and Blind SQL injection. To perform the attacks, the SQLMAP tool has been used.

NetFlow traffic has generated using DOROTHEA (DOcker-based fRamework fOr gaTHering nEtflow trAffic). NetFlow is a network protocol developed by Cisco for the collection and monitoring of network traffic flow data generated. A flow is defined as a unidirectional sequence of packets with some common properties that pass through a network device.

Datasets

The firts dataset was colleted to train the detection models (D1) and other collected using different attacks than those used in training to test the models and ensure their generalization (D2).

The datasets contain both benign and malicious traffic. All collected datasets are balanced.

The version of NetFlow used to build the datasets is 5.

    Dataset
    Aim
    Samples
    Benign-malicious
    traffic ratio




    D1
    Training
    400,003
    50%


    D2
    Test
    57,239
    50%

Infrastructure and implementation

Two sets of flow data were collected with DOROTHEA. DOROTHEA is a Docker-based framework for NetFlow data collection. It allows you to build interconnected virtual networks to generate and collect flow data using the NetFlow protocol. In DOROTHEA, network traffic packets are sent to a NetFlow generator that has a sensor ipt_netflow installed. The sensor consists of a module for the Linux kernel using Iptables, which processes the packets and converts them to NetFlow flows.

DOROTHEA is configured to use Netflow V5 and export the flow after it is inactive for 15 seconds or after the flow is active for 1800 seconds (30 minutes)

Benign traffic generation nodes simulate network traffic generated by real users, performing tasks such as searching in web browsers, sending emails, or establishing Secure Shell (SSH) connections. Such tasks run as Python scripts. Users may customize them or even incorporate their own. The network traffic is managed by a gateway that performs two main tasks. On the one hand, it routes packets to the Internet. On the other hand, it sends it to a NetFlow data generation node (this process is carried out similarly to packets received from the Internet).

The malicious traffic collected (SQLI attacks) was performed using SQLMAP. SQLMAP is a penetration tool used to automate the process of detecting and exploiting SQL injection vulnerabilities.

The attacks were executed on 16 nodes and launch SQLMAP with the parameters of the following table.

    Parameters
    Description




    '--banner','--current-user','--current-db','--hostname','--is-dba','--users','--passwords','--privileges','--roles','--dbs','--tables','--columns','--schema','--count','--dump','--comments', --schema'
    Enumerate users, password hashes, privileges, roles, databases, tables and columns


    --level=5
    Increase the probability of a false positive identification


    --risk=3
    Increase the probability of extracting data


    --random-agent
    Select the User-Agent randomly


    --batch
    Never ask for user input, use the default behavior


    --answers="follow=Y"
    Predefined answers to yes

Every node executed SQLIA on 200 victim nodes. The victim nodes had deployed a web form vulnerable to Union-type injection attacks, which was connected to the MYSQL or SQLServer database engines (50% of the victim nodes deployed MySQL and the other 50% deployed SQLServer).

The web service was accessible from ports 443 and 80, which are the ports typically used to deploy web services. The IP address space was 182.168.1.1/24 for the benign and malicious traffic-generating nodes. For victim nodes, the address space was 126.52.30.0/24. The malicious traffic in the test sets was collected under different conditions. For D1, SQLIA was performed using Union attacks on the MySQL and SQLServer databases.

However, for D2, BlindSQL SQLIAs were performed against the web form connected to a PostgreSQL database. The IP address spaces of the networks were also different from those of D1. In D2, the IP address space was 152.148.48.1/24 for benign and malicious traffic generating nodes and 140.30.20.1/24 for victim nodes.

To run the MySQL server we ran MariaDB version 10.4.12. Microsoft SQL Server 2017 Express and PostgreSQL version 13 were used.

This software tool generates simulated radar signals and creates RF datasets. The datasets can be used to develop and test detection algorithms by utilizing machine learning/deep learning techniques for the 3.5 GHz Citizens Broadband Radio Service (CBRS) or similar bands. In these bands, the primary users of the band are federal incumbent radar systems. The software tool generates radar waveforms and randomizes the radar waveform parameters. The pulse modulation types for the radar signals and their parameters are selected based on NTIA testing procedures for ESC certification, available at http://www.its.bldrdoc.gov/publications/3184.aspx. Furthermore, the tool mixes the waveforms with interference and packages them into one RF dataset file. The tool utilizes a graphical user interface (GUI) to simplify the selection of parameters and the mixing process. A reference RF dataset was generated using this software. The RF dataset is published at https://doi.org/10.18434/M32116.

SDC-Scissor tool for Cost-effective Simulation-based Test Selection in Self-driving Cars Software

This dataset provides test cases for self-driving cars with the BeamNG simulator. Check out the repository and demo video to get started.

GitHub: github.com/ChristianBirchler/sdc-scissor

This project extends the tool competition platform from the Cyber-Phisical Systems Testing Competition which was part of the SBST Workshop in 2021.

Usage

Demo

YouTube Link

Installation

The tool can either be run with Docker or locally using Poetry.

When running the simulations a working installation of BeamNG.research is required. Additionally, this simulation cannot be run in a Docker container but must run locally.

To install the application use one of the following approaches:

• Docker: docker build --tag sdc-scissor .
• Poetry: poetry install

Using the Tool

The tool can be used with the following two commands:

• Docker: docker run --volume "$(pwd)/results:/out" --rm sdc-scissor [COMMAND] [OPTIONS] (this will write all files written to /out to the local folder results)
• Poetry: poetry run python sdc-scissor.py [COMMAND] [OPTIONS]

There are multiple commands to use. For simplifying the documentation only the command and their options are described.

• Generation of tests:
  • generate-tests --out-path /path/to/store/tests
• Automated labeling of Tests:
  • label-tests --road-scenarios /path/to/tests --result-folder /path/to/store/labeled/tests
  • Note: This only works locally with BeamNG.research installed
• Model evaluation:
  • evaluate-models --dataset /path/to/train/set --save
• Split train and test data:
  • split-train-test-data --scenarios /path/to/scenarios --train-dir /path/for/train/data --test-dir /path/for/test/data --train-ratio 0.8
• Test outcome prediction:
  • predict-tests --scenarios /path/to/scenarios --classifier /path/to/model.joblib
• Evaluation based on random strategy:
  • evaluate --scenarios /path/to/test/scenarios --classifier /path/to/model.joblib

The possible parameters are always documented with --help.

Linting

The tool is verified the linters flake8 and pylint. These are automatically enabled in Visual Studio Code and can be run manually with the following commands:

poetry run flake8 .
poetry run pylint **/*.py

License

The software we developed is distributed under GNU GPL license. See the LICENSE.md file.

Contacts

Christian Birchler - Zurich University of Applied Science (ZHAW), Switzerland - birc@zhaw.ch

Nicolas Ganz - Zurich University of Applied Science (ZHAW), Switzerland - gann@zhaw.ch

Sajad Khatiri - Zurich University of Applied Science (ZHAW), Switzerland - mazr@zhaw.ch

Dr. Alessio Gambi - Passau University, Germany - alessio.gambi@uni-passau.de

Dr. Sebastiano Panichella - Zurich University of Applied Science (ZHAW), Switzerland - panc@zhaw.ch

References

• Christian Birchler, Nicolas Ganz, Sajad Khatiri, Alessio Gambi, and Sebastiano Panichella. 2022. Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor. In 2022 IEEE 29th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE.

If you use this tool in your research, please cite the following papers:

@INPROCEEDINGS{Birchler2022,
 author={Birchler, Christian and Ganz, Nicolas and Khatiri, Sajad and Gambi, Alessio, and Panichella, Sebastiano},
 booktitle={2022 IEEE 29th International Conference on Software Analysis, Evolution and Reengineering (SANER), 
 title={Cost-effective Simulationbased Test Selection in Self-driving Cars Software with SDC-Scissor}, 
 year={2022},
}

The global database testing tool market size was valued at USD 2,504.2 million in 2025 and is projected to reach USD 19,405.8 million by 2033, exhibiting a CAGR of 33.6% during the forecast period. The growth of the market is attributed to the rising demand for ensuring the accuracy and reliability of database systems, increasing adoption of cloud-based database testing tools, and growing need for automated database testing solutions to improve efficiency. Key market trends include the advancement of artificial intelligence (AI) and machine learning (ML) technologies in database testing tools, which enables automated test case generation, data validation, and performance optimization. Additionally, the increasing adoption of agile development methodologies and DevOps practices has led to the demand for continuous database testing tools that can integrate seamlessly with CI/CD pipelines. The market is also witnessing the emergence of database testing tools specifically designed for specific database types, such as NoSQL and NewSQL databases, to meet the unique testing requirements of these systems.

This file describes the contents of an archive of the 1st Competition on Software Testing (Test-Comp 2019) https://test-comp.sosy-lab.org/2019/

The competition was run by Dirk Beyer, LMU Munich, Germany. More information is available in the following article: Dirk Beyer. First International Competition on Software Testing: Test-Comp 2019. International Journal on Software Tools for Technology Transfer, 2020.

Copyright (C) Dirk Beyer https://www.sosy-lab.org/people/beyer/

SPDX-License-Identifier: CC-BY-4.0 https://spdx.org/licenses/CC-BY-4.0.html

Contents:

LICENSE.txt specifies the license README.txt this file witnessFileByHash/ This directory contains test suites (witnesses for coverage). Each witness in this directory is stored in a file whose name is the SHA2 256-bit hash of its contents followed by the filename extension .zip. The format of each test suite is described on the format web page: https://gitlab.com/sosy-lab/software/test-format A test suite contains also metadata in order to relate it to the test problem for which it was produced. witnessInfoByHash/ This directory contains for each test suite (witness) in directory witnessFileByHash/ a record in JSON format (also using the SHA2 256-bit hash of the witness as filename, with .json as filename extension) that contains the meta data. witnessListByProgramHashJSON/ For convenient access to all test suites for a certain program, this directory represents a function that maps each program (via its SHA2 256-bit hash) to a set of test suites (JSON records for test suites as described above) that the test tools have produced for that program. For each program for which test suites exist, the directory contains a JSON file (using the SHA2 256-bit hash of the program as filename, with .json as filename extension) that contains all JSON records for test suites for that program.

A similar data structure was used by SV-COMP and is described in the following article: Dirk Beyer. A Data Set of Program Invariants and Error Paths. In Proceedings of the 2019 IEEE/ACM 16th International Conference on Mining Software Repositories (MSR 2019, Montreal, Canada, May 26-27), pages 111-115, 2019. IEEE. https://doi.org/10.1109/MSR.2019.00026

Overview over archives from Test-Comp 2019 that are available at Zenodo:

https://doi.org/10.5281/zenodo.3856669 Witness store (containing the generated test suites) https://doi.org/10.5281/zenodo.3856661 Results (XML result files, log files, file mappings, HTML tables) https://doi.org/10.5281/zenodo.3856478 Test tasks, version testcomp19 https://doi.org/10.5281/zenodo.2561835 BenchExec, version 1.18

All benchmarks were executed for Test-Comp 2019, https://test-comp.sosy-lab.org/2019/ by Dirk Beyer, LMU Munich based on the components git@github.com:sosy-lab/sv-benchmarks.git testcomp19-0-g6a770a9c1 git@gitlab.com:sosy-lab/test-comp/bench-defs.git testcomp19-0-g1677027 git@github.com:sosy-lab/benchexec.git 1.18-0-gff72868

Feel free to contact me in case of questions: https://www.sosy-lab.org/people/beyer/

Software testing is one of the most crucial tasks in the typical development process. Developers are usually required to write unit test cases for the code they implement. Since this is a time-consuming task, in last years many approaches and tools for automatic test case generation — such as EvoSuite — have been introduced. Nevertheless, developers have to maintain and evolve tests to sustain the changes in the source code; therefore, having readable test cases is important to ease such a process.However, it is still not clear whether developers make an effort in writing readable unit tests. Therefore, in this paper, we conduct an explorative study comparing the readability of manually written test cases with the classes they test. Moreover, we deepen such analysis looking at the readability of automatically generated test cases. Our results suggest that developers tend to neglect the readability of test cases and that automatically generated test cases are generally even less readable than manually written ones.

Open Source Performance Testing Market Outlook

The open source performance testing market has seen a considerable rise in its market size, with global figures indicating an impressive growth trajectory. In 2023, the market was valued at approximately USD 1.2 billion and is projected to reach around USD 3.5 billion by 2032, growing at a robust compound annual growth rate (CAGR) of 12.5%. This growth is primarily driven by the increasing need for businesses to ensure seamless operational capabilities and the rising complexity of software systems that necessitate efficient performance evaluation tools. As organizations across various sectors are increasingly relying on digital platforms, the demand for performance testing tools is expected to witness a substantial surge.

The growth of the open source performance testing market can be attributed to several key factors. Firstly, the growing adoption of digital transformation initiatives across industries has led to a surge in the deployment of software solutions that require rigorous performance testing. Organizations are increasingly realizing the importance of maintaining optimal performance for their applications to enhance user experience and ensure customer satisfaction. Additionally, the cost-effectiveness of open source tools compared to proprietary software solutions has made them an attractive choice for businesses, especially small and medium enterprises (SMEs) that may have budget constraints. Moreover, the collaborative nature of open source development communities has fostered innovation and rapid advancements in performance testing tools, further propelling market growth.

Another significant factor contributing to the expansion of this market is the increasing complexity of IT infrastructure, which includes cloud computing, IoT devices, and microservices architectures. These advancements in technology have made performance testing more challenging and essential than ever before. As organizations adopt more complex and distributed systems, the need for robust performance testing solutions becomes critical to identify bottlenecks and ensure system reliability. Open source performance testing tools offer the flexibility and scalability required to cater to these evolving demands, making them a preferred choice for organizations worldwide. Furthermore, the growing emphasis on agile and DevOps methodologies has accelerated the adoption of continuous testing practices, where open source tools play a vital role in enabling seamless integration into the software development lifecycle.

From a regional perspective, the growth dynamics of the open source performance testing market show variation across different parts of the world. North America has emerged as a dominant player in this market, driven by the early adoption of advanced technologies and the presence of major technology companies. The region's robust IT infrastructure and a strong focus on technological innovation have contributed significantly to market growth. Meanwhile, the Asia Pacific region is expected to witness the highest growth rate during the forecast period. This can be attributed to the rapid digitization across various industries, increasing investments in IT infrastructure, and the rising trend of cloud adoption. Additionally, Europe also presents significant growth opportunities due to the increasing emphasis on digitalization and stringent regulatory requirements concerning software performance and security.

In the realm of software development, Test Data Generation Tools have become indispensable for ensuring the accuracy and reliability of performance testing. These tools enable developers to create realistic test data that mimics real-world scenarios, allowing for comprehensive evaluation of software applications. By simulating various user interactions and data inputs, Test Data Generation Tools help identify potential issues and optimize application performance. As organizations increasingly adopt agile and DevOps methodologies, the integration of these tools into the testing process has become crucial for maintaining high-quality software. The ability to generate diverse and complex data sets not only enhances the effectiveness of performance testing but also aids in uncovering hidden defects that could impact user experience.

Tool Type Analysis

In the open source performance testing market, the tool type segment is a critical component that defines the specific applications and functionalities of these testing platforms. Load testing t