SimpleQA

A factuality benchmark called SimpleQA that measures the ability for language models to answer short, fact-seeking questions.

  Sources

openai/simple-evals Introducing SimpleQA Measuring short-form factuality in large language models

OpenAI's o1 had the highest share of questions answered when attempted in SimpleQA benchmark in 2025. Claude-3 had the highest share of simply not attempting questions, though whether this is due to lack of data or other reasons is unknown.

oidlabs/simpleQA dataset hosted on Hugging Face and contributed by the HF Datasets community

cmriat/simpleqa dataset hosted on Hugging Face and contributed by the HF Datasets community

llamastack/simpleqa dataset hosted on Hugging Face and contributed by the HF Datasets community

SimpleQuestions is a large-scale factoid question answering dataset. It consists of 108,442 natural language questions, each paired with a corresponding fact from Freebase knowledge base. Each fact is a triple (subject, relation, object) and the answer to the question is always the object. The dataset is divided into training, validation, and test sets with 75,910, 10,845 and 21,687 questions respectively.

基于 SimpleQA 基准的最新大语言模型（LLM）性能排行榜，包含各模型的得分、发布机构、发布时间等数据。

hamishivi/SimpleQA-RLVR-noprompt dataset hosted on Hugging Face and contributed by the HF Datasets community

andreuka18/SimpleQA-1000 dataset hosted on Hugging Face and contributed by the HF Datasets community

ringos/output_Llama-3.1-8B-simpleqa-0_1000-m_generation-n_128-t_1.0-k_50-p_0.95-l_128 dataset hosted on Hugging Face and contributed by the HF Datasets community

NoahShen/simpleqa-llama3.1-8b-inst-safe-rlhf-0628-completions dataset hosted on Hugging Face and contributed by the HF Datasets community

SimpleQuestions is a dataset for simple QA, which consists of a total of 108,442 questions written in natural language by human English-speaking annotators each paired with a corresponding fact, formatted as (subject, relationship, object), that provides the answer but also a complete explanation. Fast have been extracted from the Knowledge Base Freebase (freebase.com). We randomly shuffle these questions and use 70% of them (75910) as training set, 10% as validation set (10845), and the remaining 20% as test set.

ringos/output_Mistral-Nemo-Base-2407-simpleqa-0_1000-m_generation-n_32-t_1.0-k_40-p_0.9-l_128 dataset hosted on Hugging Face and contributed by the HF Datasets community

Synthetic oracle datastore for SimpleQA. The oracle document is generated based on the problem and answer. This data is generated by Llama3.3-70B-Instruct. template = f""" You are a helpful assistant that can synthesize a Wikipedia document from a question and an answer. The document should be an actual Wikipedia article that can be helpful for answering the question. Do not directly include the question in the document. The document should contain around 150 words.

Question: {{question}} … See the full description on the dataset page: https://huggingface.co/datasets/rulins/SimpleQA-synthetic-datastore-Llama3.3-70B-Instruct.

ACG-SimpleQA

🌐 Website • 🤗 Hugging Face

中文 | English

ACG-SimpleQA is an objective knowledge question-answering dataset focused on the Chinese ACG (Animation, Comic, Game) domain, containing 4242 auto-generated carefully designed QA samples. This benchmark aims to evaluate large language models' factual capabilities in the ACG culture domain, featuring Chinese language, diversity, high quality, static answers, and easy evaluation.

  📢 Latest Updates… See the full description on the dataset page: https://huggingface.co/datasets/Papersnake/ACG-SimpleQA.

Model Card: SimpleQA Benchmark

Information from OpenAI blogpost Model Card for SimpleQAVersion: v1.0Date: October 30, 2024Authors: Jason Wei, Karina Nguyen, Hyung Won Chung, Joy Jiao, Spencer Papay, Mia Glaese, John Schulman, Liam FedusAcknowledgements: Adam Tauman Kalai

  Model Overview

SimpleQA is a factuality benchmark designed to evaluate the accuracy and reliability of language models in responding to short, fact-seeking questions. Aimed at assessing models'… See the full description on the dataset page: https://huggingface.co/datasets/MAISAAI/openai_simple_qa_test_set.

Lambent/synthetic-rag-simple-qa-4th-to-6th dataset hosted on Hugging Face and contributed by the HF Datasets community

piotr-rybak/simple-qa dataset hosted on Hugging Face and contributed by the HF Datasets community

kesitt/Wikipedia-Turkish-SimpleQA dataset hosted on Hugging Face and contributed by the HF Datasets community

Lambent/synthetic-rag-hermes-simple-qa-1st-ic dataset hosted on Hugging Face and contributed by the HF Datasets community