Webis-WikiDiscussions-18 Corpus is the output of parsing the entire set of Wikipedia talk pages. The corpus contains about six million discussions, consisting of about 20 million turns. The turns comprise around 74,000 different tags with a total of about 100,000 instances, around 7000 different shortcuts with about 400,000 instances, and around 51,000 different inline templates with about 3.3 million instances. The database has the following structure: PAGES: PAGE-ID, URL, TITLE DISCUSSIONS: DISCUSSION-ID, PAGE-ID, TITLE COMMENTS: COMMENT-ID, DISCUSSION-ID, PARENT-ID, TEXT-RAW, TEXT-CLEAN, USER TAGS: TAG-ID, COMMENT-ID, TAG-TEXT, TAG-CLASS TEMPLATES: TEMPLATE-ID, DISCUSSION-ID, TEMPLATE-TEXT SHORTCUTS: SHORTCUT-ID, COMMENT-ID, SHORTCUT-TEXT, SHORTCUT-CLASS LINKS: LINK-ID, COMMENT-ID, LINK-TEXT INLINE-TEMPLATES: IL-TEMPLATE-ID, COMMENT-ID, IL-TEMPLATE-TEXT, TYPE, DESCRIPTION

This dataset consists of a collection of Wikipedia Talk Page discussions that focus specifically on fact-checking. The discussions were selected using the language model Phi-3, which was prompted to identify and generate both a summary and a label for each discussion. BERTopic was then used to cluster the data, on full discussions, opening posts and individual comments. Before clustering, topic words from the associated Wikipedia page were masked in the texts.

Dataset Structure

The dataset is stored in JSONL (JSON Lines) format, where each line is a JSON object representing one discussion. Below is an overview of the fields included in each JSON object:

Discussion-Level Fields

• DISCUSSION-ID: A unique identifier for each discussion.
• SUMMARY: A brief summary of the discussion, capturing the main focus of the discussion, generated by Phi-3.
• PAGE-TITLE: The title of the Wikipedia page where the discussion took place.
• DISCUSSION-TITLE: The title of the discussion.
• PAGE-ID: A unique identifier for the Wikipedia Talk page.
• DISCUSSION-TOPIC: A topic classification label for the discussion, generated with BERTopic.
• OPENING-TOPIC: A topic classification label for the opening post, generated with BERTopic.
• COMMENTS: A list of comments from the discussion (described in detail below).

Comment-Level Fields

Each COMMENTS list contains dictionaries with the following fields:

• COMMENT-ID: A unique identifier for each comment.
• COMMENT-NR: The numerical order of the comment within the discussion.
• COMMENT-TOPIC: A topic classification label for the comment, generated with BERTopic.
• LEVEL: The indentation level or depth of the comment in the discussion thread (indicating whether it's a reply to another comment).
• TEXT-CLEAN: The raw text of the comment, cleaned of any special formatting.
• TEXT-MASKED: The comment text with topic words masked with [MASK] which will be recognized when embedding with BERT-based models.
• TIMESTAMP: The timestamp of when the comment was posted.
• USER: The username of the commenter.

Additional Fields

• CONTENT-WORDS: A list of topic words extracted from the associated Wikipedia page using TF-IDF.

Credits

The dataset is based on the Webis-WikiDiscussions-18 dataset