Dataset containing synthetically generated (by GPT-3.5 and GPT-4) short stories that only use a small vocabulary. Described in the following paper: https://arxiv.org/abs/2305.07759. The models referred to in the paper were trained on TinyStories-train.txt (the file tinystories-valid.txt can be used for validation loss). These models can be found on Huggingface, at roneneldan/TinyStories-1M/3M/8M/28M/33M/1Layer-21M. Additional resources: tinystories_all_data.tar.gz - contains a superset of… See the full description on the dataset page: https://huggingface.co/datasets/roneneldan/TinyStories.

TinyStories-tokenized-10k

This repository provides a pre-tokenized version of the TinyStories dataset, prepared using a custom Byte Pair Encoding (BPE) tokenizer with a vocabulary size of 10,000 tokens. This preprocessing was performed to facilitate efficient training of compact language models while maintaining high-quality language modeling performance.

  📦 Dataset Description

Source: roneneldan/TinyStories Tokenizer: Custom BPE tokenizer… See the full description on the dataset page: https://huggingface.co/datasets/KabirBakhshaei/TinyStories-tokenized-10k.