Dataset Card for GPQA

GPQA is a multiple-choice, Q&A dataset of very hard questions written and validated by experts in biology, physics, and chemistry. When attempting questions out of their own domain (e.g., a physicist answers a chemistry question), these experts get only 34% accuracy, despite spending >30m with full access to Google. We request that you do not reveal examples from this dataset in plain text or images online, to reduce the risk of leakage into foundation model… See the full description on the dataset page: https://huggingface.co/datasets/Idavidrein/gpqa.

fingertap/GPQA-Diamond dataset hosted on Hugging Face and contributed by the HF Datasets community

nikhilchandak/GPQA-diamond-free dataset hosted on Hugging Face and contributed by the HF Datasets community

nikhilchandak/gpqa-diamond-test2 dataset hosted on Hugging Face and contributed by the HF Datasets community

mkhalifa/gpqa-diamond-physics dataset hosted on Hugging Face and contributed by the HF Datasets community

GPQA Diamond Dataset

This dataset contains filtered JSONL files of human annotations on question specificity, answer uniqueness, answer matching to the ground truth for different models for the GPQA Diamond dataset.

The dataset was annotated by two human graders. It contains 198 (original size) * 2 = 396 rows as each rows is repeated twice (one for each human). A human grader given the question, actual answer and model response, has to answer whether the response matches the… See the full description on the dataset page: https://huggingface.co/datasets/nikhilchandak/gpqa-diamond-annotations.

Reasoning PRM Preference Dataset

This dataset contains reasoning traces from multiple sources (GPQA Diamond and MMLU Pro), labeled with preference information based on correctness verification.

  Dataset Description





  Overview

The dataset consists of reasoning problems and their solutions, where each example has been verified for correctness and labeled with a preference score. It combines data from two main sources:

GPQA Diamond MMLU Pro

  Data Fields… See the full description on the dataset page: https://huggingface.co/datasets/ariaattarml/verified-reasoning-o1-gpqa-mmlu-pro.

GPQA Diamond with Llama-3.1-70B-Instruct (up to 1K Samples)

This dataset contains 198 graduate-level science questions from the GPQA Diamond benchmark with up to 1000 candidate responses generated by Llama-3.1-70B-Instruct for each problem. Each response has been evaluated for correctness using a mixture of GPT-4o-mini and procedural Python code to robustly parse different answer formats, and scored by multiple reward models (scalar values) and LM judges (boolean verdicts). For more… See the full description on the dataset page: https://huggingface.co/datasets/hazyresearch/GPQA_Diamond_with_Llama_3.1_70B_Instruct_up_to_1K_Samples_v1.

aradhye/gpqa_diamond dataset hosted on Hugging Face and contributed by the HF Datasets community

In view of the significant performance improvements recently demonstrated by the Qwen3 series of models, we conducted a comprehensive evaluation of their capabilities across a range of representative benchmarks. Specifically, we evaluated the Qwen3 models on AIME2024, AIME2025, and GPQA Diamond. The prompt format used in these experiments is provided in the response files; additional details regarding the prompt design will be presented at a later time. Each set of inference experiments… See the full description on the dataset page: https://huggingface.co/datasets/Xuerui2312/Qwen3-8B-Rollout64-32k-AIME2024-AIME2025-GPQA.

Freeform Datasets

This repository contains two carefully curated datasets for evaluating large language models on human-filtered subset of popular benchmarks which are suitable for evaluation in freeform (open-ended) format. These datasets were developed as part of our paper on Answer Matching Outperforms Multiple Choice for Language Model Evaluation.

  Dataset Structure

The repository contains two splits:

  1. gpqa_diamond Split

Source: Filtered subset of… See the full description on the dataset page: https://huggingface.co/datasets/nikhilchandak/freeform-datasets.

OpenR1-Math-220k_decontaminated

Decontaminated version of open-r1/OpenR1-Math-220k - default/train

  Decontamination

Removed any questions that have an 8-gram overlap with common benchmarks: AIME 2024, AIME 2025, MATH500, GPQA Diamond, LiveCodeBench Code Generation Lite Used GitHub:huggingface/open-r1/scripts/decontaminate.py with all defaults following https://github.com/huggingface/open-r1#data-decontamination

python scripts/decontaminate.py
--dataset… See the full description on the dataset page: https://huggingface.co/datasets/notpaulmartin/OpenR1-Math-220k_decontaminated.

Answer Matching Dataset

This dataset contains a single split for human annotation analysis:

gpqa_diamond_annotations: Combined GPQA Diamond annotations from all annotators (Ameya + Nikhil)

All other evaluation files are available in the "Files and versions" tab, preserving the original directory structure.

  Directory Structure and Data Overview





  gpqa_diamond_mcq

combined_samples.jsonl samples_deepseek-r1-0528.jsonl samples_llama-4-scout.jsonl… See the full description on the dataset page: https://huggingface.co/datasets/nikhilchandak/answer-matching.

Dataset Description:

OpenScience is a multi-domain synthetic dataset designed to improve general-purpose reasoning in large language models (LLMs). The dataset contains multiple-choice question-answer pairs with detailed reasoning traces and spans across diverse scientific domains, including STEM, law, economics, and humanities. OpenScience aims to boost accuracy on advanced benchmarks such as GPQA-Diamond and MMLU-Pro via supervised finetuning or reinforcement learning. This… See the full description on the dataset page: https://huggingface.co/datasets/nvidia/OpenScience.