A. SUMMARY This dataset contains the underlying data for the Vision Zero Benchmarking website. Vision Zero is the collaborative, citywide effort to end traffic fatalities in San Francisco. The goal of this benchmarking effort is to provide context to San Francisco’s work and progress on key Vision Zero metrics alongside its peers. The Controller's Office City Performance team collaborated with the San Francisco Municipal Transportation Agency, the San Francisco Department of Public Health, the San Francisco Police Department, and other stakeholders on this project. B. HOW THE DATASET IS CREATED The Vision Zero Benchmarking website has seven major metrics. The City Performance team collected the data for each metric separately, cleaned it, and visualized it on the website. This dataset has all seven metrics and some additional underlying data. The majority of the data is available through public sources, but a few data points came from the peer cities themselves. C. UPDATE PROCESS This dataset is for historical purposes only and will not be updated. To explore more recent data, visit the source website for the relevant metrics. D. HOW TO USE THIS DATASET This dataset contains all of the Vision Zero Benchmarking metrics. Filter for the metric of interest, then explore the data. Where applicable, datasets already include a total. For example, under the Fatalities metric, the "Total Fatalities" category within the metric shows the total fatalities in that city. Any calculations should be reviewed to not double-count data with this total. E. RELATED DATASETS N/A

Benchmark for AMR Metrics based on Overt Objectives (Bamboo), the first benchmark to support empirical assessment of graph-based MR similarity metrics. Bamboo maximizes the interpretability of results by defining multiple overt objectives that range from sentence similarity objectives to stress tests that probe a metric’s robustness against meaning-altering and meaning- preserving graph transformations.

A dataset of mentions, growth rate, and total volume of the keyphrase 'Benchmark Data' over time.

An understanding of the similar and divergent metrics and methodologies underlying open government data benchmarks can reduce the risks of the potential misinterpretation and misuse of benchmarking outcomes by policymakers, politicians, and researchers. Hence, this study aims to compare the metrics and methodologies used to measure, benchmark, and rank governments' progress in open government data initiatives. Using a critical meta-analysis approach, we compare nine benchmarks with reference to meta-data, meta-methods, and meta-theories. This study finds that both existing open government data benchmarks and academic open data progress models use a great variety of metrics and methodologies, although open data impact is not usually measured. While several benchmarks’ methods have changed over time, and variables measured have been adjusted, we did not identify a similar pattern for academic open data progress models. This study contributes to open data research in three ways: 1) it reveals the strengths and weaknesses of existing open government data benchmarks and academic open data progress models; 2) it reveals that the selected open data benchmarks employ relatively similar measures as the theoretical open data progress models; and 3) it provides an updated overview of the different approaches used to measure open government data initiatives’ progress. Finally, this study offers two practical contributions: 1) it provides the basis for combining the strengths of benchmarks to create more comprehensive approaches for measuring governments’ progress in open data initiatives; and 2) it explains why particular countries are ranked in a certain way. This information is essential for governments and researchers to identify and propose effective measures to improve their open data initiatives.

Learn more about Market Research Intellect's Benchmarking Services For Transportation Rates And Logistics Performance Metrics Market Report, valued at USD 1.2 billion in 2024, and set to grow to USD 2.5 billion by 2033 with a CAGR of 9.2% (2026-2033).

A total of 12 software defect data sets from NASA were used in this study, where five data sets (part I) including CM1, JM1, KC1, KC2, and PC1 are obtained from PROMISE software engineering repository (http://promise.site.uottawa.ca/SERepository/), the other seven data sets (part II) are obtained from tera-PROMISE Repository (http://openscience.us/repo/defect/mccabehalsted/).

The Intelligence Task Ontology and Knowledge Graph (ITO) provides a comprehensive, curated model of artificial intelligence tasks, benchmarks and benchmark results, including the biomedical domain.

A dataset of mentions, growth rate, and total volume of the keyphrase 'Benchmark Saturation' over time.

DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in Digital Forensics and Incident Response Description DFIR-Metric is a comprehensive benchmark developed to assess the performance of Large Language Models (LLMs) in the field of Digital Forensics and Incident Response (DFIR), aiming to fill the gap in standardized evaluation methods. The benchmark comprises three key components: (a) MODULE I: expert-validated knowledge-based questions , (b) MODULE II: realistic forensic… See the full description on the dataset page: https://huggingface.co/datasets/Neo111x/DFIR-Metric.

Short Description This is a supporting dataset for matbench-genmetrics [docs] [repo], a set of generative materials benchmarking metrics. It contains compositional and structural fingerprints. Additional files include space group number for each structure and the "heat map" values for the empirical distribution of modified Pettifor scale-encoded (1D periodic table) values. Fingerprints The compositional (Magpie) and structural (CrystalNN) fingerprints* are produced in fingerprint_snapshot.py using mp-time-split data [docs] [repo] [figshare] and are given in comp_fingerprints.csv (132 features) and struct_fingerprints.csv (61 features), respectively. Each has an additional column in the first position, material_id, which contains the Materials Project material_id. So, in total there are 133 and 62 columns, respectively. There are 40476 entries, plus a header row with labels, so 40477 rows in total. The primary purpose of these datasets is to avoid repeating lengthy calculations each time a matbench-genmetrics benchmark is computed; thus, only the generated structures need to be featurized. The total runtime for the compositional and structural fingerprinting using 6 physical cores (12 virtual cores as determined by multiprocessing.cpu_count()) is approximately 50 minutes and 140 min, respectively. The benchmarks can be used with materials generative models such as xtal2png+Imagen. *The use of Magpie and CrystalNN featurizers are based on the coverage metric from CDVAE [repo] [paper]. A small set of dummy data for testing purposes is also included (dummy_comp_fingerprints.csv and dummy_struct_fingerprints.csv) Space Group Number The first column of space_group_number.csv is material_id, same as above, and the second column is space_group_number, as determined by the get_space_group_info() method for each pymatgen Structure object. A corresponding dummy file is provided. For the generation of this dataset, see validity_snapshot.py. Modified Pettifor Scale Each of the pymatgen Composition objects are converted to the fractional_composition counterpart and then collectively summed via np.sum() to get the fractional prevalences of periodic elements across each of the datasets. The periodic elements are then encoded in the 1D periodic table called the modified Pettifor scale. The columns of mod_petti_contributions.csv are symbol as in element symbol, mod_petti as in the modified Pettifor scale value, and contribution as in the fractional contribution to the full dataset. A dummy file is also provided for testing and debugging purposes. For the generation of this dataset, see validity_snapshot.py. The mapping dictionary is a slightly modified version of ElMD's implementation.

In this table, we’re looking at whether adding video content (including links to your video hosting platforms) could help you boost your engagement metrics, primarily the average click-th rough and click-to-open rates.

A dataset of mentions, growth rate, and total volume of the keyphrase 'Imagenet Benchmark' over time.

In this paper, we present an architecture and a formal framework to be used for systematic benchmarking of monitoring and diagnostic systems and for producing comparable performance assessments of different diagnostic technologies. The framework defines a number of standardized specifications, which include a fault catalog, a library of modular test scenarios, and a common protocol for gathering and processing diagnostic data. At the center of the framework are 13 benchmarking metric definitions. The calculation of metrics is illustrated on a probabilistic model-based diagnosis algorithm utilizing Bayesian reasoning techniques. The diagnosed system is a real-world electrical power system, namely the Advanced Diagnostics and Prognostics Testbed (ADAPT) developed and located at the NASA Ames Research Center. The proposed benchmarking approach shows how to generate realistic diagnostic data sets for large-scale, complex engineering systems, and how the generated experimental data can be used to enable “apples to apples” assessments of the effectiveness of different diagnostic and monitoring algorithms.

General

For more details and the most up-to-date information please consult our project page: https://kainmueller-lab.github.io/fisbe.

Summary

• A new dataset for neuron instance segmentation in 3d multicolor light microscopy data of fruit fly brains
  • 30 completely labeled (segmented) images
  • 71 partly labeled images
  • altogether comprising ∼600 expert-labeled neuron instances (labeling a single neuron takes between 30-60 min on average, yet a difficult one can take up to 4 hours)
• To the best of our knowledge, the first real-world benchmark dataset for instance segmentation of long thin filamentous objects
• A set of metrics and a novel ranking score for respective meaningful method benchmarking
• An evaluation of three baseline methods in terms of the above metrics and score

Abstract

Instance segmentation of neurons in volumetric light microscopy images of nervous systems enables groundbreaking research in neuroscience by facilitating joint functional and morphological analyses of neural circuits at cellular resolution. Yet said multi-neuron light microscopy data exhibits extremely challenging properties for the task of instance segmentation: Individual neurons have long-ranging, thin filamentous and widely branching morphologies, multiple neurons are tightly inter-weaved, and partial volume effects, uneven illumination and noise inherent to light microscopy severely impede local disentangling as well as long-range tracing of individual neurons. These properties reflect a current key challenge in machine learning research, namely to effectively capture long-range dependencies in the data. While respective methodological research is buzzing, to date methods are typically benchmarked on synthetic datasets. To address this gap, we release the FlyLight Instance Segmentation Benchmark (FISBe) dataset, the first publicly available multi-neuron light microscopy dataset with pixel-wise annotations. In addition, we define a set of instance segmentation metrics for benchmarking that we designed to be meaningful with regard to downstream analyses. Lastly, we provide three baselines to kick off a competition that we envision to both advance the field of machine learning regarding methodology for capturing long-range data dependencies, and facilitate scientific discovery in basic neuroscience.

Dataset documentation:

We provide a detailed documentation of our dataset, following the Datasheet for Datasets questionnaire:

>> FISBe Datasheet

Our dataset originates from the FlyLight project, where the authors released a large image collection of nervous systems of ~74,000 flies, available for download under CC BY 4.0 license.

Files

• fisbe_v1.0_{completely,partly}.zip
  • contains the image and ground truth segmentation data; there is one zarr file per sample, see below for more information on how to access zarr files.
• fisbe_v1.0_mips.zip
  • maximum intensity projections of all samples, for convenience.
• sample_list_per_split.txt
  • a simple list of all samples and the subset they are in, for convenience.
• view_data.py
  • a simple python script to visualize samples, see below for more information on how to use it.
• dim_neurons_val_and_test_sets.json
  • a list of instance ids per sample that are considered to be of low intensity/dim; can be used for extended evaluation.
• Readme.md
  • general information

How to work with the image files

Each sample consists of a single 3d MCFO image of neurons of the fruit fly.
For each image, we provide a pixel-wise instance segmentation for all separable neurons.
Each sample is stored as a separate zarr file (zarr is a file storage format for chunked, compressed, N-dimensional arrays based on an open-source specification.").
The image data ("raw") and the segmentation ("gt_instances") are stored as two arrays within a single zarr file.
The segmentation mask for each neuron is stored in a separate channel.
The order of dimensions is CZYX.

We recommend to work in a virtual environment, e.g., by using conda:

conda create -y -n flylight-env -c conda-forge python=3.9
conda activate flylight-env

How to open zarr files

1. Install the python zarr package:

   pip install zarr

2. Opened a zarr file with:
   

   import zarr
raw = zarr.open(
seg = zarr.open(

   # optional:
import numpy as np
raw_np = np.array(raw)

Zarr arrays are read lazily on-demand.
Many functions that expect numpy arrays also work with zarr arrays.
Optionally, the arrays can also explicitly be converted to numpy arrays.

How to view zarr image files

We recommend to use napari to view the image data.

1. Install napari:

   pip install "napari[all]"

2. Save the following Python script:
   

   import zarr, sys, napari

   raw = zarr.load(sys.argv[1], mode='r', path="volumes/raw")
gts = zarr.load(sys.argv[1], mode='r', path="volumes/gt_instances")

   viewer = napari.Viewer(ndisplay=3)
for idx, gt in enumerate(gts):
viewer.add_labels(
gt, rendering='translucent', blending='additive', name=f'gt_{idx}')
viewer.add_image(raw[0], colormap="red", name='raw_r', blending='additive')
viewer.add_image(raw[1], colormap="green", name='raw_g', blending='additive')
viewer.add_image(raw[2], colormap="blue", name='raw_b', blending='additive')
napari.run()

3. Execute:

   python view_data.py 

Metrics

• S: Average of avF1 and C
• avF1: Average F1 Score
• C: Average ground truth coverage
• clDice_TP: Average true positives clDice
• FS: Number of false splits
• FM: Number of false merges
• tp: Relative number of true positives

For more information on our selected metrics and formal definitions please see our paper.

Baseline

To showcase the FISBe dataset together with our selection of metrics, we provide evaluation results for three baseline methods, namely PatchPerPix (ppp), Flood Filling Networks (FFN) and a non-learnt application-specific color clustering from Duan et al..
For detailed information on the methods and the quantitative results please see our paper.

License

The FlyLight Instance Segmentation Benchmark (FISBe) dataset is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Citation

If you use FISBe in your research, please use the following BibTeX entry:

@misc{mais2024fisbe,
 title =    {FISBe: A real-world benchmark dataset for instance
         segmentation of long-range thin filamentous structures},
 author =    {Lisa Mais and Peter Hirsch and Claire Managan and Ramya
         Kandarpa and Josef Lorenz Rumberger and Annika Reinke and Lena
         Maier-Hein and Gudrun Ihrke and Dagmar Kainmueller},
 year =     2024,
 eprint =    {2404.00130},
 archivePrefix ={arXiv},
 primaryClass = {cs.CV}
}

Acknowledgments

We thank Aljoscha Nern for providing unpublished MCFO images as well as Geoffrey W. Meissner and the entire FlyLight Project Team for valuable
discussions.
P.H., L.M. and D.K. were supported by the HHMI Janelia Visiting Scientist Program.
This work was co-funded by Helmholtz Imaging.

Changelog

There have been no changes to the dataset so far.
All future change will be listed on the changelog page.

Contributing

If you would like to contribute, have encountered any issues or have any suggestions, please open an issue for the FISBe dataset in the accompanying github repository.

All contributions are welcome!

This dataset contains raw, unprocessed data files pertaining to the management tool 'Benchmarking'. The data originates from five distinct sources, each reflecting different facets of the tool's prominence and usage over time. Files preserve the original metrics and temporal granularity before any comparative normalization or harmonization. Data Sources & File Details: Google Trends File (Prefix: GT_): Metric: Relative Search Interest (RSI) Index (0-100 scale). Keywords Used: "benchmarking" + "benchmarking management" Time Period: January 2004 - January 2025 (Native Monthly Resolution). Scope: Global Web Search, broad categorization. Extraction Date: Data extracted January 2025. Notes: Index relative to peak interest within the period for these terms. Reflects public/professional search interest trends. Based on probabilistic sampling. Source URL: Google Trends Query Google Books Ngram Viewer File (Prefix: GB_): Metric: Annual Relative Frequency (% of total n-grams in the corpus). Keywords Used: Benchmarking Time Period: 1950 - 2022 (Annual Resolution). Corpus: English. Parameters: Case Insensitive OFF, Smoothing 0. Extraction Date: Data extracted January 2025. Notes: Reflects term usage frequency in Google's digitized book corpus. Subject to corpus limitations (English bias, coverage). Source URL: Ngram Viewer Query Crossref.org File (Prefix: CR_): Metric: Absolute count of publications per month matching keywords. Keywords Used: "benchmarking" AND ("process" OR "management" OR "performance" OR "best practices" OR "implementation" OR "approach" OR "evaluation" OR "methodology") Time Period: 1950 - 2025 (Queried for monthly counts based on publication date metadata). Search Fields: Title, Abstract. Extraction Date: Data extracted January 2025. Notes: Reflects volume of relevant academic publications indexed by Crossref. Deduplicated using DOIs; records without DOIs omitted. Source URL: Crossref Search Query Bain & Co. Survey - Usability File (Prefix: BU_): Metric: Original Percentage (%) of executives reporting tool usage. Tool Names/Years Included: Benchmarking (1993, 1996, 1999, 2000, 2002, 2004, 2006, 2008, 2010, 2012, 2014, 2017). Respondent Profile: CEOs, CFOs, COOs, other senior leaders; global, multi-sector. Source: Bain & Company Management Tools & Trends publications (Rigby D., Bilodeau B., et al., various years: 1994, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015, 2017). Note: Tool not included in the 2022 survey data. Data Compilation Period: July 2024 - January 2025. Notes: Data points correspond to specific survey years. Sample sizes: 1993/500; 1996/784; 1999/475; 2000/214; 2002/708; 2004/960; 2006/1221; 2008/1430; 2010/1230; 2012/1208; 2014/1067; 2017/1268. Bain & Co. Survey - Satisfaction File (Prefix: BS_): Metric: Original Average Satisfaction Score (Scale 0-5). Tool Names/Years Included: Benchmarking (1993, 1996, 1999, 2000, 2002, 2004, 2006, 2008, 2010, 2012, 2014, 2017). Respondent Profile: CEOs, CFOs, COOs, other senior leaders; global, multi-sector. Source: Bain & Company Management Tools & Trends publications (Rigby D., Bilodeau B., et al., various years: 1994, 2001, 2003, 2005, 2007, 2009, 2011, 2013, 2015, 2017). Note: Tool not included in the 2022 survey data. Data Compilation Period: July 2024 - January 2025. Notes: Data points correspond to specific survey years. Sample sizes: 1993/500; 1996/784; 1999/475; 2000/214; 2002/708; 2004/960; 2006/1221; 2008/1430; 2010/1230; 2012/1208; 2014/1067; 2017/1268. Reflects subjective executive perception of utility. File Naming Convention: Files generally follow the pattern: PREFIX_Tool.csv, where the PREFIX indicates the data source: GT_: Google Trends GB_: Google Books Ngram CR_: Crossref.org (Count Data for this Raw Dataset) BU_: Bain & Company Survey (Usability) BS_: Bain & Company Survey (Satisfaction) The essential identification comes from the PREFIX and the Tool Name segment. This dataset resides within the 'Management Tool Source Data (Raw Extracts)' Dataverse.

Benchmarking Software Market Outlook

The global market size for benchmarking software was valued at approximately USD 1.5 billion in 2023 and is projected to reach USD 4.8 billion by 2032, growing at a compound annual growth rate (CAGR) of 14.2% during the forecast period. This growth is primarily driven by the increasing need for businesses to enhance their operational efficiency and competitive edge through comprehensive performance analysis and strategic planning.

One of the primary growth factors driving the benchmarking software market is the rapid digital transformation across various industries. Organizations are increasingly adopting advanced technologies to streamline operations, reduce costs, and improve decision-making processes. Benchmarking software provides companies with critical insights into their performance relative to industry standards and competitors, enabling them to identify areas for improvement and optimize their resources effectively. This demand for strategic business intelligence tools is expected to continue propelling the market forward.

Another significant driver is the growing emphasis on data-driven decision-making. In today's competitive business environment, relying on intuition or traditional methods is no longer sufficient. Companies are leveraging data analytics to gain actionable insights into their performance metrics. Benchmarking software facilitates the collection, analysis, and interpretation of vast amounts of data, helping businesses make informed decisions and stay ahead of the competition. This trend is particularly prominent in sectors such as BFSI, healthcare, and IT and telecommunications.

The increasing complexity of business operations and the need for continuous improvement also contribute to the market's growth. Organizations are continually seeking ways to enhance their processes, reduce inefficiencies, and achieve higher productivity. Benchmarking software enables them to compare their performance against industry leaders and best practices, identify gaps, and implement targeted strategies for improvement. This continuous improvement culture is gaining traction across various sectors, further driving the demand for benchmarking solutions.

In terms of regional outlook, North America holds a significant share of the benchmarking software market, owing to the presence of numerous technology-driven enterprises and a high adoption rate of advanced software solutions. The region's well-established IT infrastructure and the growing focus on digital transformation initiatives are key factors contributing to market growth. Europe is also expected to witness substantial growth, driven by the increasing need for performance optimization and regulatory compliance. The Asia Pacific region is anticipated to register the highest CAGR during the forecast period, fueled by rapid industrialization, increasing investments in technology, and the rising adoption of benchmarking software by SMEs.

In the realm of performance analytics, Business Dashboard Software plays a pivotal role by providing organizations with a visual representation of their key performance indicators (KPIs) and metrics. These dashboards enable businesses to monitor their performance in real-time, facilitating quick decision-making and strategic planning. By integrating benchmarking software with business dashboards, companies can gain a comprehensive view of their performance relative to industry standards, allowing them to identify areas for improvement and optimize their operations. This synergy of tools not only enhances data visualization but also empowers organizations to drive continuous improvement and maintain a competitive edge in the market.

Component Analysis

The benchmarking software market can be segmented into software and services. The software segment holds the largest market share, driven by the widespread adoption of advanced IT solutions across various industries. Benchmarking software encompasses a wide range of tools and platforms designed to facilitate performance analysis, data visualization, and strategic planning. These solutions enable organizations to collect, analyze, and interpret data from multiple sources, providing valuable insights into their performance metrics and helping them make data-driven decisions.

Within the software segment, cloud-based benchmarking solutions are gaining significant traction. The flexibility, scalability, and cost-ef

Technological advancements in the Benchmarking Services for Transportation Rates and Logistics Performance Metrics industry are shaping the future market landscape. The report evaluates innovation-driven growth and how emerging technologies are transforming industry practices, offering a comprehensive outlook on future opportunities and market potential.

Dataset Card for BOOM (Benchmark of Observability Metrics)

  Dataset Summary

BOOM (Benchmark of Observability Metrics) is a large-scale, real-world time series dataset designed for evaluating models on forecasting tasks in complex observability environments. Composed of real-world metrics data collected from Datadog, a leading observability platform, the benchmark captures the irregularity, structural complexity, and heavy-tailed statistics typical of production… See the full description on the dataset page: https://huggingface.co/datasets/Datadog/BOOM.

This repo contains a complete replication package, including raw data and scripts for the statistical analysis, for the paper "Ghost Echoes Revealed: Benchmarking Maintainability Metrics and Machine Learning Predictions Against Human Assessments" submitted to the industry track of the 40th International Conference on Software Maintenance and Evolution (ICSME), Flagstaff, AZ, USA, Oct 6-11, 2024.

• • uc1_maintainability_prediction.ipynb: A Notebook for Use Case 1 - Maintainability Prediction.
  • uc2_liability_prediction.ipynb: A Notebook for Use Case 2 - Liability Prediction.
• maintainability_data.csv: The dataset covering the 404 open-source files from the Maintainability Dataset (Bertrand et al., 2020)
  • Majority vote ground truth labels from Schnappinger et al. (2020)
  • Low-level code metrics from Bertrand et al. (2023)
  • Code Health
  • SonarQube output, i.e., TD Ratio and TD Time
  • Microsoft Maintainability index provided by MetricsTree

• Schnappinger et al., A Software Maintainability Dataset, 10.6084/m9.figshare.12801215, 2020.
• Schnappinger et al., Defining a Software Maintainability Dataset: Collecting, Aggregating and Analysing Expert Evaluations of Software Maintainability, in Proc. of the 36th International Conference on Software Maintenance and Evolution, pp. 278–289, 2020.
• S. Bertrand et al., Replication and Extension of Schnappinger’s Study on Human-level Ordinal Maintainability Prediction Based on Static Code Metrics, in Proc. of the 27th International Conference on Evaluation and Assessment in Software Engineering, pp. 241–246, 2023.

Dataset containing AI model performance metrics and benchmarking results