Facebook
TwitterAttribution-ShareAlike 4.0 (CC BY-SA 4.0)https://creativecommons.org/licenses/by-sa/4.0/
License information was derived automatically
Pre-parsed English and French Wikipedia articles, extracted using the Wikimedia Enterprise Snapshot API.
This dataset contains all articles of the English and French language editions of Wikipedia, pre-parsed and output as structured data with a consistent schema. The dataset is provided in Parquet format, optimized for high-performance analytical queries and efficient storage. This version uses a unified, pinned schema across all files, making it compatible with DuckDB, pandas, Polars, and Apache Spark out of the box.
New in this dataset:
referenceneed and referencerisk, signaling where information may not be sufficiently backed by sources.The dataset is built as part of the Structured Contents initiative and based on Wikimedia Enterprise HTML snapshots. This updated version includes pre-parsed Wikipedia abstracts, short descriptions, main image, infoboxes, article sections, tables, references, citations, lists, article images, and more.
For updates, follow the Wikimedia Enterprise blog and MediaWiki Quarterly software updates.
We welcome feedback to help refine and improve this dataset. Please share suggestions or issues on the Wikimedia Enterprise Meta-Wiki discussion page or on this dataset’s Kaggle discussion page.
The contents of this dataset are collectively written and curated by a global volunteer community. Please see the Attribution Information section for licensing and attribution guidance.
The dataset in its structured form is generally helpful for a wide variety of tasks, including model development, pre-training, alignment, fine-tuning, retrieval-augmented generation, updating, testing, and benchmarking.
We would love to hear more about your use cases.
enfrThere is one language edition for English and one for French. They encompass national and cultural variations of spelling, vocabulary, grammar, and usage. Within a Wikipedia language edition, no national variety is officially preferred over others. The rule of thumb is that the conventions of a particular language variety should be followed consistently within a given article.
As part of this beta Wikimedia Enterprise Structured Contents initiative, the team is expanding coverage of language editions. To request a specific language edition, please contact us on the Meta-Wiki talk page and explain your use case.
Each row represents one Wikipedia artic...
Facebook
TwitterWikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump (https://dumps.wikimedia.org/) with one split per language. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.).
To use this dataset:
import tensorflow_datasets as tfds
ds = tfds.load('wikipedia', split='train')
for ex in ds.take(4):
print(ex)
See the guide for more informations on tensorflow_datasets.
Facebook
Twitterhttps://choosealicense.com/licenses/gfdl/https://choosealicense.com/licenses/gfdl/
this is a subset of the wikimedia/wikipedia dataset code for creating this dataset : from datasets import load_dataset, Dataset from sentence_transformers import SentenceTransformer model = SentenceTransformer("mixedbread-ai/mxbai-embed-large-v1")
dataset = load_dataset( "wikimedia/wikipedia", "20231101.en", split="train", streaming=True )
from tqdm import tqdm data = Dataset.from_dict({}) for i, entry in… See the full description on the dataset page: https://huggingface.co/datasets/not-lain/wikipedia-small-3000-embedded.
Facebook
TwitterDataset Card for Speech Wikimedia
Dataset Summary
The Speech Wikimedia Dataset is a compilation of audiofiles with transcriptions extracted from wikimedia commons that is licensed for academic and commercial usage under CC and Public domain. It includes 2,000+ hours of transcribed speech in different languages with a diverse set of speakers. Each audiofile should have one or more transcriptions in different languages.
Transcription languages… See the full description on the dataset page: https://huggingface.co/datasets/MLCommons/speech-wikimedia.
Facebook
Twitterhttps://brightdata.com/licensehttps://brightdata.com/license
Bright Data's Wikipedia dataset gives you structured, ready-to-use article data from English Wikipedia, one of the largest open knowledge bases on the web. It includes over 9.1 million article records with 11 data fields each, ready for AI training, search and research. Each record includes the article URL and ID, title, table of contents, full raw text, cataloged text organized by section, images, categories, references, external links and see-also links. AI and machine learning teams, researchers, search and knowledge graph builders and data analysts use the Wikipedia dataset to train and fine-tune large language models, build retrieval-augmented generation (RAG) systems and knowledge graphs, run entity and topic research, and analyze how knowledge is linked and cited. Refreshed quarterly. Pricing starts at $250 per 100K records. You can buy the full dataset or a filtered subset by category, title, keywords, article length or any other field, as a one-time purchase or on a recurring refresh schedule. A free data sample is available. Data formats: JSON, NDJSON, CSV, XLSX and Parquet. Delivery options: Amazon S3, Google Cloud Storage, Azure, Snowflake, SFTP, webhook or direct download. All data is collected from publicly available sources in line with GDPR and CCPA.
Facebook
TwitterAttribution-ShareAlike 3.0 (CC BY-SA 3.0)https://creativecommons.org/licenses/by-sa/3.0/
License information was derived automatically
wikimedia/wikipedia — Wikipedia dataset containing cleaned articles of all languages. The dataset is built from the Wikipedia dumps ( with one subset per language, each containing a single train split. Each example contains the content of one full Wikipedia article with cleaning to strip markdown… On Linux: load_dataset('wikimedia/wikipedia'). 1,670 Hugging Face likes (#5 in this directory).
Facebook
TwitterAttribution-ShareAlike 4.0 (CC BY-SA 4.0)https://creativecommons.org/licenses/by-sa/4.0/
License information was derived automatically
Wikipedia dumps contain a tremendous amount of markup. WikiMedia Text is a hybrid of markdown and HTML, making it very difficult to use. Wikipedia, however, is an extremely valuable dataset. I wanted a more usable format.
This dataset includes ~40MB JSON files, each of which contains a collection of Wikipedia articles. Each article element in the JSON contains only 3 keys: an ID number, the title of the article, and the text of the article. Each article has been "flattened" to occupy a single plain text string. This makes it easier for humans to read, as opposed to the markup version. It also makes it easier for NLP tasks. You will have much less cleanup to do.
Each file looks like this:
[
{
"id": "17279752",
"text": "Hawthorne Road was a cricket and football ground in Bootle in England...",
"title": "Hawthorne Road"
}
]
Absolutely all thanks goes to Wikipedia! Everyone who helped build and fund Wikipedia, and everyone who has contributed their time and expertise to the content of Wikipedia
I was inspired to create this dataset because I needed an offline encyclopedia for an Artificial General Intelligence project I'm working on.
Facebook
TwitterAttribution-ShareAlike 3.0 (CC BY-SA 3.0)https://creativecommons.org/licenses/by-sa/3.0/
License information was derived automatically
Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump ( with one split per language. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.).
Facebook
TwitterMIT Licensehttps://opensource.org/licenses/MIT
License information was derived automatically
A corpus created from the Hugging Face Wikipedia dataset (https://huggingface.co/datasets/wikipedia). The preprocessing and the creation of this corpus are done using the text_to_sentences method of Blingfire. The details can be found in the following notebook:
https://www.kaggle.com/code/emmermarcell/create-a-wikipedia-corpus
Facebook
TwitterThis dataset was created by Debarshi Chanda
Facebook
TwitterCC0 1.0 Universal Public Domain Dedicationhttps://creativecommons.org/publicdomain/zero/1.0/
License information was derived automatically
We provide a corpus of discussion comments from English Wikipedia talk pages. Comments are grouped into different files by year. Comments are generated by computing diffs over the full revision history and extracting the content added for each revision. See our wiki for documentation of the schema and our research paper for documentation on the data collection and processing methodology.
Facebook
TwitterAttribution-ShareAlike 3.0 (CC BY-SA 3.0)https://creativecommons.org/licenses/by-sa/3.0/
License information was derived automatically
Dataset Card for Wikimedia Wikisource
Dataset Summary
Wikisource dataset containing cleaned articles of all languages. The dataset is built from the Wikisource dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikisource text with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump… See the full description on the dataset page: https://huggingface.co/datasets/wikimedia/wikisource.
Facebook
TwitterAttribution-ShareAlike 3.0 (CC BY-SA 3.0)https://creativecommons.org/licenses/by-sa/3.0/
License information was derived automatically
Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump with one split per language.
Facebook
TwitterAttribution 4.0 (CC BY 4.0)https://creativecommons.org/licenses/by/4.0/
License information was derived automatically
Movies-related articles extracted from Wikipedia. For all articles, the figures and tables have been filtered out, as well as the categories and "see also" sections. The article structure, and particularly the sub-titles and paragraphs are kept in these datasets Movies The Wikipedia Movies dataset consists of 100,371 articles describing various movies. Each article may consist of text passages describing the plot, cast, production, reception, soundtrack, and more.
Facebook
TwitterAttribution-ShareAlike 3.0 (CC BY-SA 3.0)https://creativecommons.org/licenses/by-sa/3.0/
License information was derived automatically
Wikipedia contains a huge amount of data that could be used to do certain task, especially in Natural Language Processing (NLP), either to classify text, build a checkpoint or a model, or do certain exploration about the text itself. Most of the Large Language Model (LLM) leverage this huge dataset to build, improve, or test it's model. This dataset contains almost all the pages from wikipedia Indonesia that have been preprocessed before in order to clean up some Wikipedia formatting that's irrelevant to overall data. Some text is not fully perfect, some remaining formatting still exist and if you think you could clean it better, I'll be happy to provide the raw data. Wikipedia also statet that due to it's free usage, we should not scrap the web, or use automatic crawler to get the data. They also statet that downloading through mirror link sure help them a lot by reducing bandwith. By using this dataset, anyone could benefit, either for Wikipedia, for us scientist, or for the general good.
Text Classification Text Exploration Build Tokenizer Translation Question Answering Text generation Other NLP Task
| Feature | Description |
|---|---|
| id | unique page id |
| parent_id | the parent id of the page. Some page are subpage of other page |
| timestamp | the creation date of the page. available in %Y-%m-%d %H:%M:%S format |
| hash | unique hash that differentiate page based on it's content |
| title | title of the page |
| text | the content of the page that's been preprocessed to clear wikipedia formatting |
| category | category of a page. some value is missing due to uncomplete preprocessing |
The original data is contained in this link.
Wikipedia
Facebook
TwitterAttribution-ShareAlike 3.0 (CC BY-SA 3.0)https://creativecommons.org/licenses/by-sa/3.0/
License information was derived automatically
range3/wikipedia-ja-20230101
This dataset consists of a parquet file from the wikipedia dataset with only Japanese data extracted. It is generated by the following python code. このデータセットは、wikipediaデータセットの日本語データのみを抽出したparquetファイルで構成されます。以下のpythonコードによって生成しています。 import datasets dss = datasets.load_dataset( "wikipedia", language="ja", date="20230101", beam_runner="DirectRunner", )
for split,ds in dss.items():… See the full description on the dataset page: https://huggingface.co/datasets/range3/wikipedia-ja-20230101.
Facebook
TwitterWikimedia Commons is a media file repository making available public domain and freely-licensed educational media content (images, sound and video clips) to everyone, in their own language. It acts as a common repository for the various projects of the Wikimedia Foundation, but you do not need to belong to one of those projects to use media hosted here.
Facebook
TwitterAttribution-ShareAlike 4.0 (CC BY-SA 4.0)https://creativecommons.org/licenses/by-sa/4.0/
License information was derived automatically
Wikipedia Multilingual Fragments (WMF) is a multilingual corpus of clean text fragments extracted from 340 different Wikipedia language editions using the official MediaWiki API.
The dataset contains over 373,000 plain-text fragments, each between 500 and 1500 characters, collected by randomly sampling Wikipedia articles in encyclopedic namespaces. For each language, up to 2 million characters were collected, discarding articles that were too short or contained excessive formatting. Fragments were cleaned by removing section headers, citations, LaTeX expressions, and redundant whitespace.
The repository includes:
wmf.csv: a unified CSV file with one row per text fragment. Each row includes metadata such as:
language code
article ID and title
character count of the excerpt
timestamp of extraction
a flag indicating whether the excerpt was truncated
the cleaned text itself
chunks/: 8,212 plain-text files, each containing exactly 50,000 characters. These were generated by concatenating all excerpts per language and splitting them into fixed-length, non-overlapping segments. Only complete chunks were retained. Languages that could not produce at least one full chunk were excluded, resulting in a final count of 325 languages in this stage.
analysis/n_grams/: character-level n-gram frequency tables (n=1 to 20) for each language, based on the cleaned text.
analysis/tfidf/: word- and character-based TF, IDF, and TF-IDF scores per language, depending on the script (e.g., word-based for English, character-based for Chinese, Japanese, etc.).
scripts/: Python scripts to reproduce the dataset and analyses:
generate_fragments_from_wikipedia.py: collects and filters plain-text excerpts from Wikipedia.
merge_language_csvs.py: merges individual per-language CSVs into a unified dataset (wmf.csv).
chunk_creation.py: creates fixed-length (50,000 characters) text chunks from the full dataset, removing invisible Unicode characters and discarding incomplete segments.
n_grams_calculator.py: computes character-level n-gram frequencies (1 to 20) per language.
tfidf_calculator.py: calculates TF, IDF, and TF-IDF scores per language using language-aware tokenization.
This dataset is designed for:
multilingual language modeling
benchmarking tokenization strategies
cross-linguistic comparison
low-resource NLP
statistical and linguistic analysis
Facebook
TwitterWorked for 22sI've created a modern Wikidata-inspired HTML design with a responsive layout, search bar, example knowledge items, a custom logo concept, and a light/dark theme toggle.The design follows ideas from MediaWiki's modern Vector interface, but it is an independent prototype—not an official Wikidata redesign.### Preview of the design### W### WikidataTHE FREE KNOWLEDGE BASEModern UI### Knowledge, connected.Explore items, statements, and identifiers in a clean, mobile-friendly interface.ItemsPeople and placesStatementsProperties and valuesLinked dataConnected knowledge### Download the HTML filewikidata-modern.htmlHTML · 7 KB · CSS and JavaScript includedDownload Wikidata Modern HTML Copy filename### How to use it* Download the HTML file above.* Open it in Chrome or another web browser.* To edit the code, open it in GitHub or a code editor.* To publish it, upload wikidata-modern.html to your website hosting or rename it to index.html.Note: The built-in search filters sample items only. Live Wikidata data, account login, and editing require additional integration.
Facebook
TwitterAttribution 4.0 (CC BY 4.0)https://creativecommons.org/licenses/by/4.0/
License information was derived automatically
## Overview
Wikipedia is a dataset for object detection tasks - it contains UI Elements annotations for 5,522 images.
## Getting Started
You can download this dataset for use within your own projects, or fork it into a workspace on Roboflow to create your own model.
## License
This dataset is available under the [CC BY 4.0 license](https://creativecommons.org/licenses/CC BY 4.0).
Facebook
TwitterAttribution-ShareAlike 4.0 (CC BY-SA 4.0)https://creativecommons.org/licenses/by-sa/4.0/
License information was derived automatically
Pre-parsed English and French Wikipedia articles, extracted using the Wikimedia Enterprise Snapshot API.
This dataset contains all articles of the English and French language editions of Wikipedia, pre-parsed and output as structured data with a consistent schema. The dataset is provided in Parquet format, optimized for high-performance analytical queries and efficient storage. This version uses a unified, pinned schema across all files, making it compatible with DuckDB, pandas, Polars, and Apache Spark out of the box.
New in this dataset:
referenceneed and referencerisk, signaling where information may not be sufficiently backed by sources.The dataset is built as part of the Structured Contents initiative and based on Wikimedia Enterprise HTML snapshots. This updated version includes pre-parsed Wikipedia abstracts, short descriptions, main image, infoboxes, article sections, tables, references, citations, lists, article images, and more.
For updates, follow the Wikimedia Enterprise blog and MediaWiki Quarterly software updates.
We welcome feedback to help refine and improve this dataset. Please share suggestions or issues on the Wikimedia Enterprise Meta-Wiki discussion page or on this dataset’s Kaggle discussion page.
The contents of this dataset are collectively written and curated by a global volunteer community. Please see the Attribution Information section for licensing and attribution guidance.
The dataset in its structured form is generally helpful for a wide variety of tasks, including model development, pre-training, alignment, fine-tuning, retrieval-augmented generation, updating, testing, and benchmarking.
We would love to hear more about your use cases.
enfrThere is one language edition for English and one for French. They encompass national and cultural variations of spelling, vocabulary, grammar, and usage. Within a Wikipedia language edition, no national variety is officially preferred over others. The rule of thumb is that the conventions of a particular language variety should be followed consistently within a given article.
As part of this beta Wikimedia Enterprise Structured Contents initiative, the team is expanding coverage of language editions. To request a specific language edition, please contact us on the Meta-Wiki talk page and explain your use case.
Each row represents one Wikipedia artic...