GAIA dataset

GAIA is a benchmark which aims at evaluating next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc). We added gating to prevent bots from scraping the dataset. Please do not reshare the validation or test set in a crawlable format.

  Data and leaderboard

GAIA is made of more than 450 non-trivial question with an unambiguous answer, requiring different levels of tooling and autonomy to solve. It… See the full description on the dataset page: https://huggingface.co/datasets/gaia-benchmark/GAIA.

We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92% vs. 15% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA’s philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system’s capability to exhibit similar robustness as the average human does on such questions. Using GAIA’s methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board accessible at https://huggingface.co/gaia-benchmark.

Dataset Card for "resultspublic"

More Information needed

Contains the scored submissions on the GAIA benchmark, validation set only. To avoid contamination of GAIA and leakage of the answers, please do not reshare the current dataset without any gating system or outside of the HF hub.

GAIA Benchmark Subset Model Card

This dataset is a subset of the GAIA benchmark, containing 44 web-search-based questions from the validation set. It evaluates multiple AI models on their ability to retrieve and process real-time information using web search and browser tools. Performance metrics include success indicators and detailed reports for each model. A comparative chart summarizing the results will be provided separately.

  Benchmark Results

VizieR Online Data Catalog: Gaia RVS benchmark stars. II.(Caffau E.+, 2024)

Gaia Inc. Company Profile, Opportunities, Challenges and Risk (SWOT, PESTLE and Value Chain); Corporate and ESG Strategies; Competitive Intelligence; Financial KPI’s; Operational KPI’s; Recent Trends: “ Read More

VizieR Online Data Catalog: Gaia FGK benchmark stars v3(Soubiran C.+, 2024)

Real incomes are routinely compared internationally using methods that "correct" for deviations from purchasing power parity. The most widely used of these is the Geary method which, though theoretically suspect, underlies the Penn World Table. This paper provides a theoretical foundation for the Geary method which I call the GAIA ( "Geary-Allen International Accounts" ) system. I show that the Geary method is exact when preferences are non-homothetic Leontief and, more generally, gives a ( possibly poor ) approximation to the GAIA benchmark. An empirical application suggests that both it and other widely used methods underestimate the degree of international inequality.