Context

Wikipedia dumps contain a tremendous amount of markup. WikiMedia Text is a hybrid of markdown and HTML, making it very difficult to use. Wikipedia, however, is an extremely valuable dataset. I wanted a more usable format.

• Blog post about this project
• GitHub repo for this project

Content

This dataset includes ~40MB JSON files, each of which contains a collection of Wikipedia articles. Each article element in the JSON contains only 3 keys: an ID number, the title of the article, and the text of the article. Each article has been "flattened" to occupy a single plain text string. This makes it easier for humans to read, as opposed to the markup version. It also makes it easier for NLP tasks. You will have much less cleanup to do.

Each file looks like this:

[
 {
 "id": "17279752",
 "text": "Hawthorne Road was a cricket and football ground in Bootle in England...",
 "title": "Hawthorne Road"
 }
]

Acknowledgements

Absolutely all thanks goes to Wikipedia! Everyone who helped build and fund Wikipedia, and everyone who has contributed their time and expertise to the content of Wikipedia

Inspiration

I was inspired to create this dataset because I needed an offline encyclopedia for an Artificial General Intelligence project I'm working on.

simple-wikipedia

Processed, text-only dump of the Simple Wikipedia (English). Contains 23,886,673 words.

Wikipedia-PT

  Dataset Summary

The Portuguese portion of the Wikipedia dataset.

  Supported Tasks and Leaderboards

The dataset is generally used for Language Modeling.

  Languages

Portuguese

  Dataset Structure





  Data Instances

An example looks as follows: { 'text': 'Abril é o quarto mês...' }

  Data Fields

text (str): Text content of the article.

  Data Splits

All configurations contain a single train split.… See the full description on the dataset page: https://huggingface.co/datasets/TucanoBR/wikipedia-PT.

We provide a corpus of discussion comments from English Wikipedia talk pages. Comments are grouped into different files by year. Comments are generated by computing diffs over the full revision history and extracting the content added for each revision. See our wiki for documentation of the schema and our research paper for documentation on the data collection and processing methodology.

range3/wikipedia-ja-20230101

This dataset consists of a parquet file from the wikipedia dataset with only Japanese data extracted. It is generated by the following python code. このデータセットは、wikipediaデータセットの日本語データのみを抽出したparquetファイルで構成されます。以下のpythonコードによって生成しています。 import datasets dss = datasets.load_dataset( "wikipedia", language="ja", date="20230101", beam_runner="DirectRunner", )

for split,ds in dss.items(): ds.to_parquet(f"wikipedia-ja-20230101/{split}.parquet")

Dataset

This dataset was created by Geetansh Patle

Released under MIT

Contents

Dataset Card for Wikimedia Wikipedia

User interaction networks of Wikipedia of 28 different languages. Nodes (orininal wikipedia user IDs) represent users of the Wikipedia, and an edge from user A to user B denotes that user A wrote a message on the talk page of user B at a certain timestamp.

More info: http://yfiua.github.io/academic/2016/02/14/wiki-talk-datasets.html

For each existing Wikipedia language edition, the dataset contains a classification of the articles that represent its associated cultural context, i.e. all concepts and entities related to the language and to the territories where it is spoken (places, traditions, language, politics, agriculture, biographies, events, etcetera.).For each article, the dataset contains a rich set of context-related features, including geolocation, ISO codes, wikidata properties related to the language or to the corresponding country or territories, as well as related categories, among many other metadata. Other general article features are additional included, such as the number of edits and number of pageviews.The methodology employed to classify articles through machine learning is described in: Wikipedia Cultural Diversity Observatory: Cultural Context Content Methodology Miquel-Ribé, M., & Laniado, D. (2018). Wikipedia Culture Gap: Quantifying Content Imbalances Across 40 Language Editions. Frontiers in Physics.The uses of the dataset are several but we want to highlight three: 1) Wikipedia Culture Gap assessment and overall improvement of the cultural diversity, 2) Academic research in the Digital Humanities field, and 3) User-generated Content based technologies.You can read more at wcdo.wmflabs.org.

Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump (https://dumps.wikimedia.org/) with one split per language. Each example contains the content of one full Wikipedia article with cleanin...

Wikipediaを用いた日本語の固有表現抽出データセット

GitHub: https://github.com/stockmarkteam/ner-wikipedia-dataset/ LICENSE: CC-BY-SA 3.0

Developed by Stockmark Inc.

Wikipedia page titles are great topic tags and hashtags for NLP. Wikipedia has the added benefit of annotating some of the title in categories.

For example, if you want to create a tag list of film stars there is no better source than Wikipedia.

For example the tag/title Anarchism has a lot of meta information associated with it.

Anarchism ['Anarchism| ', 'Anti-capitalism', 'Anti-fascism', 'Economic ideologies', 'Far-left politics', 'Left-wing politics', 'Libertarian socialism', 'Libertarianism', 'Political culture', 'Political ideologies', 'Political movements', 'Social theories', 'Socialism']

This dataset contains a row for every (wiki, user, month) that contains a count of all 'revisions' saved and a count of those revisions that were 'archived' when the page was deleted. For more information, see https://meta.wikimedia.org/wiki/Research:Monthly_wikimedia_editor_activity_dataset Fields: · wiki -- The dbname of the wiki in question ("enwiki" == English Wikipedia, "commonswiki" == Commons) · month -- YYYYMM · user_id -- The user's identifier in the local wiki · user_name -- The user name in the local wiki (from the 'user' table) · user_registration -- The recorded registration date for the user in the 'user' table · archived -- The count of deleted revisions saved in this month by this user · revisions -- The count of all revisions saved in this month by this user (archived or not) · attached_method -- The method by which this user attached this account to their global account

Dataset Card for Wikimedia Wikisource

Dataset Summary

Wikisource dataset containing cleaned articles of all languages. The dataset is built from the Wikisource dumps (https://dumps.wikimedia.org/) with one subset per language, each c...

Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump (https://dumps.wikimedia.org/) with one split per language. Each example contains the content of one full Wikipedia article with cleanin...

Regularly published dataset of PageRank scores for Wikidata entities. The underlying link graph is formed by a union of all links accross all Wikipedia language editions. Computation is performed by Andreas Thalhammer with 'danker' available at https://github.com/athalhammer/danker .

Wikipedia dataset containing cleaned articles of all languages. The datasets are built from the Wikipedia dump (https://dumps.wikimedia.org/) with one split per language. Each example contains the content of one full Wikipedia article with cleanin...

Dataset SummaryThe Triple-to-Text Alignment dataset aligns Knowledge Graph (KG) triples from Wikidata with diverse, real-world textual sources extracted from the web. Unlike previous datasets that rely primarily on Wikipedia text, this dataset provides a broader range of writing styles, tones, and structures by leveraging Wikidata references from various sources such as news articles, government reports, and scientific literature. Large language models (LLMs) were used to extract and validate text spans corresponding to KG triples, ensuring high-quality alignments. The dataset can be used for training and evaluating relation extraction (RE) and knowledge graph construction systems.Data FieldsEach row in the dataset consists of the following fields:subject (str): The subject entity of the knowledge graph triple.rel (str): The relation that connects the subject and object.object (str): The object entity of the knowledge graph triple.text (str): A natural language sentence that entails the given triple.validation (str): LLM-based validation results, including:Fluent Sentence(s): TRUE/FALSESubject mentioned in Text: TRUE/FALSERelation mentioned in Text: TRUE/FALSEObject mentioned in Text: TRUE/FALSEFact Entailed By Text: TRUE/FALSEFinal Answer: TRUE/FALSEreference_url (str): URL of the web source from which the text was extracted.subj_qid (str): Wikidata QID for the subject entity.rel_id (str): Wikidata Property ID for the relation.obj_qid (str): Wikidata QID for the object entity.Dataset CreationThe dataset was created through the following process:1. Triple-Reference Sampling and ExtractionAll relations from Wikidata were extracted using SPARQL queries.A sample of KG triples with associated reference URLs was collected for each relation.2. Domain Analysis and Web ScrapingURLs were grouped by domain, and sampled pages were analyzed to determine their primary language.English-language web pages were scraped and processed to extract plaintext content.3. LLM-Based Text Span Selection and ValidationLLMs were used to identify text spans from web content that correspond to KG triples.A Chain-of-Thought (CoT) prompting method was applied to validate whether the extracted text entailed the triple.The validation process included checking for fluency, subject mention, relation mention, object mention, and final entailment.4. Final Dataset Statistics12.5K Wikidata relations were analyzed, leading to 3.3M triple-reference pairs.After filtering for English content, 458K triple-web content pairs were processed with LLMs.80.5K validated triple-text alignments were included in the final dataset.

Full English Wikipedia (plain text)

Unsupervised text corpus of all 4,366,845 articles on the English Wikipedia.

• ? M tokens
• ? unique words
• 22GB (uncompressed)

Extracted from the Wikipedia dumps using an open source tool, using an AWS EC2 (cloud VM).

Data is remarkably clean and uniform.

There is also a dataset available for the Simple English Wikipedia, generated in the same manner as this dataset. It is much more compact at just 120MB, and easier to work with: https://www.kaggle.com/datasets/ffatty/plain-text-wikipedia-simpleenglish/

Format:

• Each article's title appears before the content.
• Articles are plain text; they are stripped of all Wiki formatting syntax, including font styles, citations, links, etc.
• Articles are concatenated into .txt files of ≤ 1MB each.

Random example excerpt:

(a portion of 1 file; 2 articles, concatenated in place)

Peter Burroughs

Peter Burroughs (born 27 January 1947) is a British television and film actor and the director of Willow Management.

Burroughs initially ran a shop in his village at Yaxley, Cambridgeshire.

His first dramatic role was that of the character ""Branic"" in the 1979 television series "The Legend of King Arthur". He played roles in Hollywood movies such as "Flash Gordon", "The Dark Crystal", "Labyrinth", and "The Hitchhiker's Guide to the Galaxy". He portrayed a bank goblin in the Harry Potter series ("Harry Potter and the Philosopher's Stone" and "Harry Potter and the Deathly Hallows – Part 2").


Eddi McKee

Eddi McKee is a fictional character from the BBC medical drama "Holby City", played by actress Sarah-Jane Potts. She first appeared in the thirteenth series episode "Rescue Me", broadcast on 7 June 2011. Eddi was a Senior Nurse at Holby City Hospital. 

Eddi is portrayed as being straight-talking, no-nonsense, loyal and compassionate. Eddi believed the nurses did the most important job in the hospital and she had very forthright opinions. Exploration of the character's backstory began when her younger brother, Liam (Luke Tittensor), was introduced and it was revealed that Eddi had run away from home to escape their alcoholic mother. Potts pointed out that Eddi herself liked to drink to deal with her many issues. The character had a relationship with Luc Hemingway (Joseph Millson), which proved popular with viewers. Eddi also became involved with locum consultant Max Schneider (John Light) and developed an addiction to painkillers.

MaralGPT/persian-wikipedia dataset hosted on Hugging Face and contributed by the HF Datasets community