🍷 FineWeb

15 trillion tokens of the finest data the 🌐 web has to offer

  What is it?

The 🍷 FineWeb dataset consists of more than 18.5T tokens (originally 15T tokens) of cleaned and deduplicated english web data from CommonCrawl. The data processing pipeline is optimized for LLM performance and ran on the 🏭 datatrove library, our large scale data processing library. 🍷 FineWeb was originally meant to be a fully open replication of 🦅 RefinedWeb, with a release… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb.

🥂 FineWeb2

A sparkling update with 1000s of languages

  What is it?

This is the second iteration of the popular 🍷 FineWeb dataset, bringing high quality pretraining data to over 1000 🗣️ languages. The 🥂 FineWeb2 dataset is fully reproducible, available under the permissive ODC-By 1.0 license and extensively validated through hundreds of ablation experiments. In particular, on the set of 9 diverse languages we used to guide our processing decisions, 🥂 FineWeb2… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-2.

📚 FineWeb-Edu

1.3 trillion tokens of the finest educational data the 🌐 web has to offer

Paper: https://arxiv.org/abs/2406.17557

  What is it?

📚 FineWeb-Edu dataset consists of 1.3T tokens and 5.4T tokens (FineWeb-Edu-score-2) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version. To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by LLama3-70B-Instruct. We then… See the full description on the dataset page: https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu.

Ultra-FineWeb

📜 Technical Report

  📚 Introduction

Ultra-FineWeb is a large-scale, high-quality, and efficiently-filtered dataset. We use the proposed efficient verification-based high-quality filtering pipeline to the FineWeb and Chinese FineWeb datasets (source data from Chinese FineWeb-edu-v2, which includes IndustryCorpus2, MiChao, WuDao, SkyPile, WanJuan, ChineseWebText, TeleChat, and CCI3), resulting in the creation of higher-quality Ultra-FineWeb-en… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/Ultra-FineWeb.

Ornaments/fineweb dataset hosted on Hugging Face and contributed by the HF Datasets community

We recommend you to use the improved version Fineweb-edu-chinese-v2.1 !

  Chinese Fineweb Edu Dataset V2     [中文]  [English]

[OpenCSG Community] [👾github] [wechat] [Twitter]

📖Technical Report Chinese Fineweb Edu Dataset V2 is a comprehensive upgrade of the original Chinese Fineweb Edu, designed and optimized for natural language processing (NLP) tasks in the education sector. This high-quality Chinese pretraining dataset has undergone significant… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/chinese-fineweb-edu-v2.

⭐ Please download the dataset from here.

  PRIMUS: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training





  🤗 Primus-FineWeb

The Primus-FineWeb dataset is constructed by filtering cybersecurity-related text from FineWeb, a refined version of Common Crawl. We began by leveraging Primus-Seed, a high-quality dataset of manually curated cybersecurity text, as positive samples. We then sampled ten times the amount of data from FineWeb as negative samples… See the full description on the dataset page: https://huggingface.co/datasets/trend-cybertron/Primus-FineWeb.

📚 fineweb-pro

ArXiv | Models | Code fineweb-pro is refined from fineweb(350BT sample) using the ProX refining framework. It contains about 100B high quality tokens, ready for general language model pre-training.

  License

fineweb-pro is based on fineweb, which is made available under an ODC-By 1.0 license; users should also abide by the CommonCrawl ToU: https://commoncrawl.org/terms-of-use/. We do not alter the license of any of the underlying data.… See the full description on the dataset page: https://huggingface.co/datasets/gair-prox/FineWeb-pro.

FineWeb-C: Educational content in many languages, labelled by the community

Multilingual data is better together!

Note: We are not actively working on this project anymore. You can continue to contribute annotations and we'll occasionally refresh the exported data.

  What is this?

FineWeb-C is a collaborative, community-driven project that expands upon the FineWeb2 dataset. The goal is to create high-quality educational content annotations across hundreds of… See the full description on the dataset page: https://huggingface.co/datasets/data-is-better-together/fineweb-c.

Occiglot Fineweb v1.0

We present a more mature version of the multilingual Occiglot Fineweb corpus. In this early form, the dataset contains roughly 430M heavily cleaned documents from 10 languages. Occiglot Fineweb builds on our existing collection of curated datasets and pre-filtered web data. Subsequently, all documents were filtered with language-specific derivatives of the fine-web processing pipeline and different levels of depuplicated. We provide the data at 3 levels of… See the full description on the dataset page: https://huggingface.co/datasets/occiglot/occiglot-fineweb-v1.0.

ivnle/fineweb dataset hosted on Hugging Face and contributed by the HF Datasets community

Annotations for 📚 FineWeb-Edu classifier

This dataset contains the annotations used for training 📚 FineWeb-Edu educational quality classifier. We prompt Llama-3-70B-Instruct to score web pages from 🍷 FineWeb based on their educational value. Note: the dataset contains the FineWeb text sample, the prompt (using the first 1000 characters of the text sample) and the scores but it doesn't contain the full Llama 3 generation.

dododo1234/fineweb dataset hosted on Hugging Face and contributed by the HF Datasets community

OpenCoder Dataset

The OpenCoder dataset is composed of the following datasets:

opc-sft-stage1: the sft data used for opencoder sft-stage1 opc-sft-stage2: the sft data used for opencoder sft-stage2 opc-annealing-corpus: the synthetic data & algorithmic corpus used for opencoder annealing opc-fineweb-code-corpus: the code-related page recalled from fineweb <-- you are here opc-fineweb-math-corpus: the math-related page recalled from finewebrefineCode-code-corpus-meta: the meta-data… See the full description on the dataset page: https://huggingface.co/datasets/OpenCoder-LLM/opc-fineweb-code-corpus.

This is a subset of the Fineweb dataset trimmed down to approximately one billion tokens. No special frills. We sampled from the 10 billion token subset to create this one.

A subset of FineWeb-Edu randomly sampled from the whole dataset of around 1B gpt2 tokens. This dataset is created for illustration purpose in retrieval-scaling. Please do not distribute.

Massive Genre-Audience Augment Fineweb-Edu Corpus

This dataset is a synthetic pretraining corpus described in paper Reformulation for Pretraining Data Augmentation.

Overview of synthesis framework. Our method expands the original corpus through a two-stage synthesis process. Each document is reformulated to 5 new documents, achieving 3.9× token number expansion while maintaining diversity through massive (genre, audience) pairs.

We build MGACorpus based on SmolLM Corpus… See the full description on the dataset page: https://huggingface.co/datasets/ByteDance-Seed/mga-fineweb-edu.

Pre-shuffled fineweb-edu dataset

Ultra FineWeb EDU

High-Quality Educational Content from Ultra-FineWeb Filtered for Maximum Educational Value

  📚 Overview

Ultra FineWeb EDU is a premium educational dataset created by applying advanced educational content filtering to the exceptional Ultra-FineWeb dataset. This work builds directly upon two foundational achievements: the rigorous data curation methodology of Ultra-FineWeb and the sophisticated educational classification capabilities of the… See the full description on the dataset page: https://huggingface.co/datasets/ProCreations/Ultra-FineWeb-EDU.

Chinese Fineweb Edu Dataset V2.1 [中文] [English]

[OpenCSG Community] [👾github] [wechat] [Twitter]

📖Technical Report The Chinese Fineweb Edu Dataset V2.1 is an enhanced version of the V2 dataset, designed specifically for natural language processing (NLP) tasks in the education sector. This version introduces two new data sources, map-cc and opencsg-cc, and retains data with scores ranging from 2 to 3. The dataset entries are organized into different folders… See the full description on the dataset page: https://huggingface.co/datasets/opencsg/Fineweb-Edu-Chinese-V2.1.