The Webis TLDR Corpus (2017) consists of approximately 4 Million content-summary pairs extracted for Abstractive Summarization, from the Reddit dataset for the years 2006-2016. This corpus is first of its kind from the social media domain in English and has been created to compensate the lack of variety in the datasets used for abstractive summarization research using deep learning models.

This corpus contains preprocessed posts from the Reddit dataset, suitable for abstractive summarization using deep learning. The format is a json file where each line is a JSON object representing a post. The schema of each post is shown below:

• author: string (nullable = true)
• body: string (nullable = true)
• normalizedBody: string (nullable = true)
• content: string (nullable = true)
• content_len: long (nullable = true)
• summary: string (nullable = true)
• summary_len: long (nullable = true)
• id: string (nullable = true)
• subreddit: string (nullable = true)
• subreddit_id: string (nullable = true)
• title: string (nullable = true)

Specifically, the content and summary fields can be directly used as inputs to a deep learning model (e.g. Sequence to Sequence model ). The dataset consists of 3,848,330 posts with an average length of 270 words for content, and 28 words for the summary. The dataset is a combination of both the Submissions and Comments merged on the common schema. As a result, most of the comments which do not belong to any submission have null as their title.

Note : This corpus does not contain a separate test set. Thus it is up to the users to divide the corpus into appropriate training, validation and test sets.

Summarize TL;DR Filtered Dataset

This is the version of the dataset used in https://arxiv.org/abs/2009.01325. If starting a new project we would recommend using https://huggingface.co/datasets/openai/summarize_from_feedback. For more information see https://github.com/openai/summarize-from-feedback and for the original TL;DR dataset see https://huggingface.co/datasets/webis/tldr-17.

This dataset is derived from this reddit dataset

We only keep the id and content of the posts in the subreddit "relationships" with length more than 50 characters. Out intend is to use for fine tuning text generation models.