A comprehensive Wikipedia dataset containing 100,000 pages with 28.9 million links, collected using breadth-first search crawling algorithm. This dataset includes complete page metadata, link relationships, and a network graph representation suitable for network analysis, graph algorithms, NLP research, and machine learning applications.

Dataset Overview

• Total Pages: 100,000
• Total Links: 28,855,738 (directed edges)
• Average Words per Page: 3,531
• Language: English (en.wikipedia.org)
• Collection Method: BFS (Breadth-First Search) crawling, depth 5
• Data Quality Score: 99.76/100

Files Description

1. pages_export.csv

Complete page metadata including: - id: Unique page ID - title: Page title - language: Language code (en) - content_length: Content length in characters - word_count: Word count - categories: JSON array of categories - infobox: JSON object of infobox data - created_at: Timestamp - url: Full Wikipedia URL

Size: ~70 MB | Rows: 100,000

2. links_export.csv

Complete link graph with URLs: - id: Unique link ID - source_title: Source page title - target_title: Target page title - language: Language code - position: Link position on page - depth: Crawl depth where link was discovered - created_at: Timestamp - source_url: Full source page URL - target_url: Full target page URL

Size: ~4.5 GB | Rows: 28,855,738

3. graph.json

Network graph in JSON format: - nodes: Array of node objects with id field - edges: Array of edge objects with source and target fields

Size: ~2.1 GB | Edges: 28,855,738

Data Quality

• Content Coverage: 99.99% (99,992 pages have quality content)
• Link Quality: 99.22%
• Uniqueness: 100% (all links are unique)
• Content Quality: 100% (average 3,531 words per page)
• Duplicate Pages: Minimal (cleaned)
• Self-Links: 4,326 (removed)
• Data Validation: ✅ All entries validated and cleaned

Use Cases

1. Network Analysis: Study Wikipedia link structure and page connectivity
2. Graph Algorithms: Test shortest path, centrality, community detection algorithms
3. NLP Research: Analyze Wikipedia content, categories, and relationships
4. Machine Learning: Train models on Wikipedia link prediction
5. Knowledge Graph: Build knowledge graphs from Wikipedia structure
6. PageRank: Implement and test PageRank algorithms
7. Recommendation Systems: Build content recommendation systems

Collection Methodology

1. Seed Selection: Started with 5 Wikipedia pages
2. Crawling: BFS algorithm, depth 5
3. Rate Limiting: Balanced (0.82 pages/second)
4. Parallel Processing: Optimized concurrent workers
5. Caching: HTML content cached for efficiency
6. Validation: All data validated and deduplicated
7. Quality Control: Automated quality checks and cleaning

Technical Details

• Database: SQLite with WAL mode
• Crawl Duration: ~29 hours
• Crawl Rate: 0.82 pages/second
• Checkpoint System: Resume-capable crawling
• Data Cleaning: Automated duplicate removal and quality checks

Wikipediaを用いた日本語の固有表現抽出データセット

GitHub: https://github.com/stockmarkteam/ner-wikipedia-dataset/ LICENSE: CC-BY-SA 3.0

Developed by Stockmark Inc.

This dataset comprises raw data extracted from Wikipedia, encompassing various types of content including articles, metadata, and user interactions. The dataset is in its unprocessed form, providing an excellent opportunity for data enthusiasts and professionals to engage in data cleaning and preprocessing tasks. It is ideal for those looking to practice and enhance their data cleaning skills, as well as for researchers and developers who require a rich and diverse corpus for natural language processing (NLP) projects.

Dataset

This dataset was created by JAYAPRAKASHPONDY

Released under CC0: Public Domain

Contents

A subset of articles extracted from the French Wikipedia XML dump. Data published here include 5 different categories : Economy (Economie), History (Histoire), Informatics (Informatique), Health (Medecine) and Law (Droit). The Wikipedia dump was downloaded on November 8, 2016 from https://dumps.wikimedia.org/. Each article is a xml file extracted from the dump and save as UTF8 plain text. The characteristics of dataset is :

• Economy : 44'876 articles
• History : 92'041 articles
• Informatics : 25'408 articles
• Health : 22'143 articles
• Law : 9'964 articles

Dataset Card for Wikimedia Wikipedia

Kensho Derived Wikimedia Dataset

Wikipedia, the free encyclopedia, and Wikidata, the free knowledge base, are crowd-sourced projects supported by the Wikimedia Foundation. Wikipedia is nearly 20 years old and recently added its six millionth article in English. Wikidata, its younger machine-readable sister project, was created in 2012 but has been growing rapidly and currently contains more than 75 million items.

These projects contribute to the Wikimedia Foundation's mission of empowering people to develop and disseminate educational content under a free license. They are also heavily utilized by computer science research groups, especially those interested in natural language processing (NLP). The Wikimedia Foundation periodically releases snapshots of the raw data backing these projects, but these are in a variety of formats and were not designed for use in NLP research. In the Kensho R&D group, we spend a lot of time downloading, parsing, and experimenting with this raw data. The Kensho Derived Wikimedia Dataset (KDWD) is a condensed subset of the raw Wikimedia data in a form that we find helpful for NLP work. The KDWD has a CC BY-SA 3.0 license, so feel free to use it in your work too.

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4301984%2F972e4157b97efe8c2c5ea17c983b1504%2Fkdwd_header_logos_2.jpg?generation=1580510520532141&alt=media" alt="">

This particular release consists of two main components - a link annotated corpus of English Wikipedia pages and a compact sample of the Wikidata knowledge base. We version the KDWD using the raw Wikimedia snapshot dates. The version string for this dataset is kdwd_enwiki_20191201_wikidata_20191202 indicating that this KDWD was built from the English Wikipedia snapshot from 2019 December 1 and the Wikidata snapshot from 2019 December 2. Below we describe these components in more detail.

Example Notebooks

Dive right in by checking out some of our example notebooks:

• Introduction to the KDWD Wikipedia sample
• Introduction to the KDWD Wikidata sample
• Entity aliases and disambiguation candidates from anchor link statistics

Updates / Changelog

• initial release 2020-01-31

File Summary

• Wikipedia
  • page.csv (page metadata and Wikipedia-to-Wikidata mapping)
  • link_annotated_text.jsonl (plaintext of Wikipedia pages with link offsets)
• Wikidata
  • item.csv (item labels and descriptions in English)
  • item_aliases.csv (item aliases in English)
  • property.csv (property labels and descriptions in English)
  • property_aliases.csv (property aliases in English)
  • statements.csv (truthy qpq statements)

Three Layers of Data

The KDWD is three connected layers of data. The base layer is a plain text English Wikipedia corpus, the middle layer annotates the corpus by indicating which text spans are links, and the top layer connects the link text spans to items in Wikidata. Below we'll describe these layers in more detail.

https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F4301984%2F19663d43bade0e92f578255f6e0d9dcd%2Fkensho_wiki_triple_layer.svg?generation=1580347573004185&alt=media" alt="">

Wikipedia Sample

The first part of the KDWD is derived from Wikipedia. In order to create a corpus of mostly natural text, we restrict our English Wikipedia page sample to those that:

• are in the (Main/Article) namespace
• are not redirect pages
• are not disambiguation pages
• a...

We provide a corpus of discussion comments from English Wikipedia talk pages. Comments are grouped into different files by year. Comments are generated by computing diffs over the full revision history and extracting the content added for each revision. See our wiki for documentation of the schema and our research paper for documentation on the data collection and processing methodology.

Wikipedia-PT

  Dataset Summary

The Portuguese portion of the Wikipedia dataset.

  Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

  Languages

Portuguese

  Dataset Structure





  Data Instances

An example looks as follows: { 'text': 'Abril é o quarto mês...' }

  Data Fields

text (str): Text content of the article.

  Data Splits

All configurations contain a single train split.… See the full description on the dataset page: https://huggingface.co/datasets/TucanoBR/wikipedia-PT.

Wikipedia Featured Articles multimodal dataset

Overview

• This is a multimodal dataset of featured articles containing 5,638 articles and 57,454 images.
• Its superset of good articles is also hosted on Kaggle. It has six times more entries although with a little worse quality.

It contains the text of an article and also all the images from that article along with metadata such as image titles and descriptions. From Wikipedia, we selected featured articles, which are just a small subset of all available ones, because they are manually reviewed and protected from edits. Thus it's the best theoretical quality human editors on Wikipedia can offer.

You can find more details in "Image Recommendation for Wikipedia Articles" thesis.

Dataset structure

The high-level structure of the dataset is as follows:

.
+-- page1 
|  +-- text.json 
|  +-- img 
|    +-- meta.json
+-- page2 
|  +-- text.json 
|  +-- img 
|    +-- meta.json
: 
+-- pageN 
|  +-- text.json 
|  +-- img 
|    +-- meta.json

text.JSON Schema

Below you see an example of how data is stored:

{
 "title": "Naval Battle of Guadalcanal",
 "id": 405411,
 "url": "https://en.wikipedia.org/wiki/Naval_Battle_of_Guadalcanal",
 "html": "... 

...", "wikitext": "... The '''Naval Battle of Guadalcanal''', sometimes referred to as ...", }

Please note that @html and @wikitext properties represent the same information in different formats, so just choose the one which is easier to parse in your circumstances.

meta.JSON Schema

{
 "img_meta": [
  {
   "filename": "702105f83a2aa0d2a89447be6b61c624.jpg",
   "title": "IronbottomSound.jpg",
   "parsed_title": "ironbottom sound",
   "url": "https://en.wikipedia.org/wiki/File%3AIronbottomSound.jpg",
   "is_icon": False,
   "on_commons": True,
   "description": "A U.S. destroyer steams up what later became known as ...",
   "caption": "Ironbottom Sound. The majority of the warship surface ...",
   "headings": ['Naval Battle of Guadalcanal', 'First Naval Battle of Guadalcanal', ...],
   "features": ['4.8618264', '0.49436468', '7.0841103', '2.7377882', '2.1305492', ...],
   },
   ...
  ]
}

Collection method

Data was collected by fetching featured articles text&image content with pywikibot library and then parsing out a lot of additional metadata from HTML pages from Wikipedia and Commons.

• Kaggle Data Collection Notebook...

Three corpora in different domains extracted from Wikipedia. For all datasets, the figures and tables have been filtered out, as well as the categories and "see also" sections. The article structure, and particularly the sub-titles and paragraphs are kept in these datasets Wines Wikipedia wines dataset consists of 1635 articles from the wine domain. The extracted dataset consists of a non-trivial mixture of articles, including different wine categories, brands, wineries, grape types, and more. The ground-truth recommendations were crafted by a human sommelier, which annotated 92 source articles with ~10 ground-truth recommendations for each sample. Examples for ground-truth expert-based recommendations are Dom Pérignon - Moët & Chandon Pinot Meunier - Chardonnay Movies The Wikipedia movies dataset consists of 100385 articles describing different movies. The movies' articles may consist of text passages describing the plot, cast, production, reception, soundtrack, and more. For this dataset, we have extracted a test set of ground truth annotations for 50 source articles using the "BestSimilar" database. Each source articles is associated with a list of ${\scriptsize \sim}12$ most similar movies. Examples for ground-truth expert-based recommendations are Schindler's List - The Pianist Lion King - The Jungle Book Video games The Wikipedia video games dataset consists of 21,935 articles reviewing video games from all genres and consoles. Each article may consist of a different combination of sections, including summary, gameplay, plot, production, etc. Examples for ground-truth expert-based recommendations are: Grand Theft Auto - Mafia Burnout Paradise - Forza Horizon 3

Plain text of Wikipedia

Dataset Description Size Example use (python) Data fields Notes on data formatting

License Aknowledgements Citation

  Dataset Description

This dataset is a plain text version of pages from wikipedia.org spaces for several languages (English, German, French, Spanish, Italian). The text is without HTML tags nor wiki templates. It just includes markdown syntax for headers, lists and tables. See Notes on data formatting for more details. It was… See the full description on the dataset page: https://huggingface.co/datasets/OpenLLM-France/wikipedia.

Wikidata is a free, CC0-licensed knowledge base operated by the Wikimedia Foundation. All public APIs (Wikibase REST, MediaWiki Action, SPARQL Query Service, Linked Data interface, Recent Changes event stream) are free of charge with no paid tiers. Heavy or commercial users are expected to follow the User-Agent and maxlag policies, mirror data via the database dumps, or contract with Wikimedia Enterprise for commercial-grade access.

📚 Overview: This dataset provides a compact and efficient way to explore the massive "Wikipedia Structured Contents" dataset by Wikimedia Foundation, which consists of 38 large JSONL files (each ~2.5GB). Loading these directly in Kaggle or Colab is impractical due to resource constraints. This file index solves that problem.

🔍 What’s Inside: This dataset includes a single JSONL file named wiki_structured_dataset_navigator.jsonl that contains metadata for every file in the English portion of the Wikimedia dataset.

Each line in the JSONL file is a JSON object with the following fields: - file_name: the actual filename in the source dataset (e.g., enwiki_namespace_0_0.jsonl) - file_index: the numeric row index of the file - name: the Wikipedia article title or identifier - url: a link to the full article on Wikipedia - description: a short description or abstract of the article (when available)

🛠 Use Case: Use this dataset to search by keyword, article name, or description to find which specific files from the full Wikimedia dataset contain the topics you're interested in. You can then download only the relevant file(s) instead of the entire dataset.

⚡️ Benefits: - Lightweight (~MBs vs. GBs) - Easy to load and search - Great for indexing, previewing, and subsetting the Wikimedia dataset - Saves time, bandwidth, and compute resources

📎 Example Usage (Python): ```python import kagglehub import json import pandas as pd import numpy as np import os from tqdm import tqdm from datetime import datetime import re

def read_jsonl(file_path, max_records=None): data = [] with open(file_path, 'r', encoding='utf-8') as f: for i, line in enumerate(tqdm(f)): if max_records and i >= max_records: break data.append(json.loads(line)) return data

file_path = kagglehub.dataset_download("mehranism/wikimedia-structured-dataset-navigator-jsonl",path="wiki_structured_dataset_navigator.jsonl") data = read_jsonl(file_path) print(f"Successfully loaded {len(data)} records")

df = pd.DataFrame(data) print(f"Dataset shape: {df.shape}") print(" Columns in the dataset:") for col in df.columns: print(f"- {col}")

This dataset is perfect for developers working on:
- Retrieval-Augmented Generation (RAG)
- Large Language Model (LLM) fine-tuning
- Search and filtering pipelines
- Academic research on structured Wikipedia content

💡 Tip:
Pair this index with the original [Wikipedia Structured Contents dataset](https://www.kaggle.com/datasets/wikimedia-foundation/wikipedia-structured-contents) for full article access.

📃 Format:
- File: `wiki_structured_dataset_navigator.jsonl`
- Format: JSON Lines (1 object per line)
- Encoding: UTF-8

---

### **Tags**

wikipedia, wikimedia, jsonl, structured-data, search-index, metadata, file-catalog, dataset-index, large-language-models, machine-learning ```

Licensing

CC0: Public Domain Dedication

(Recommended for open indexing tools with no sensitive data.)

simple wikipedia

the 'simple' split of Wikipedia, from Sept 1 2023. The train split contains about 65M tokens, Pulled via: dataset = load_dataset( "wikipedia", language="simple", date="20230901", beam_runner="DirectRunner" )

  stats





  train split

general info

0 id… See the full description on the dataset page: https://huggingface.co/datasets/pszemraj/simple_wikipedia.

This dataset contains information about politicians from DBpedia, a crowd-sourced community effort to extract structured information from Wikipedia and make this information available on the Web. Some important information about people that is available on DBpedia are name, gender, nationality, occupation, birth date, death date, profession and for many politicians also the political party they belong to. This dataset is based on the English DBpedia dump from October 2015 and documents the temporal evolution of the hyperlink network that articles about politicians formed on Wikipedia between 2001 and 2016 every month. Wikipedia maintains revisions for each article to keep track of the changes over time. The first revision of each month was used to construct the hyperlink network between articles about politicians.

Dataset

This dataset was created by Threatthriver

Contents

Investigation of Semi-Structured Tables: WikiTableQuestions

A Dataset of Complex Questions on Semi-Structured Wikipedia Tables

By [source]

About this dataset

The WikiTableQuestions dataset poses complex questions about the contents of semi-structured Wikipedia tables. Beyond merely testing a model's knowledge retrieval capabilities, these questions require an understanding of both the natural language used and the structure of the table itself in order to provide a correct answer. This makes the dataset an excellent testing ground for AI models that aim to replicate or exceed human-level intelligence

More Datasets

For more datasets, click here.

Featured Notebooks

• 🚨 Your notebook can be here! 🚨!

How to use the dataset

In order to use the WikiTableQuestions dataset, you will need to first understand the structure of the dataset. The dataset is comprised of two types of files: questions and answers. The questions are in natural language, and are designed to test a model's ability to understand the table structure, understand the natural language question, and reason about the answer. The answers are in a list format, and provide additional information about each table that can be used to answer the questions.

To start working with the WikiTableQuestions dataset, you will need to download both the questions and answers files. Once you have downloaded both files, you can begin working with the dataset by loading it into a pandas dataframe. From there, you can begin exploring the data and developing your own models for answering the questions.

Happy Kaggling!

Research Ideas

• The WikiTableQuestions dataset can be used to train a model to answer complex questions about semi-structured Wikipedia tables.

• The WikiTableQuestions dataset can be used to train a model to understand the structure of semi-structured Wikipedia tables.

• The WikiTableQuestions dataset can be used to train a model to understand the natural language questions and reason about the answers

Acknowledgements

If you use this dataset in your research, please credit the original authors.

Data Source

License

License: CC0 1.0 Universal (CC0 1.0) - Public Domain Dedication No Copyright - You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission. See Other Information.

Columns

File: 0.csv

File: 1.csv

File: 10.csv

File: 11.csv

File: 12.csv

File: 14.csv

File: 15.csv

File: 17.csv

File: 18.csv

Acknowledgements

If you use this dataset in your research, please credit the original authors. If you use this dataset in your research, please credit .

Wikipedia is a massive repository of human knowledge. The largest edition, the English Wikipedia, contains over 65.5 million pages, including 7.17 million articles (excluding redirects). Connecting this vast network are 1.63 billion unique page-to-page links. Based on an analysis of this dataset, the most cited articles on the English Wikipedia were identified. When considering what these most cited articles in Wikipedia might be, we can assume that prominent historical topics like “United… See the full description on the dataset page: https://huggingface.co/datasets/lewoniewski/most-cited-wikipedia-articles.

How many people actually use Wikipedia each month? This data answers that by counting unique devices that visit English Wikipedia. If you check Wikipedia once a day for a month, that's one device counted. If you check it on your phone and your laptop, that's two. Wikimedia tracks this using cookies and your IP address, so they can roughly tell if it's you coming back or someone new. It's a straightforward measure of Wikipedia's audience over time.