by Ronen Eldan
TinyStories is a dataset of synthetic short stories written with a deliberately small vocabulary. It is meant for training and evaluating small language models.
DatasetFree
Step 1: Open the dataset page on Hugging Face and read the README and the linked paper.
Step 2: Download the training and validation text files, or the V2 training file for the larger version.
Step 3: Train a small model on the training file and use the validation file to track loss.
Step 4: Use the evaluation prompts file to test the trained model as the paper does.
Step 5: Use the full archive if you need the metadata and generation prompts for each story.
Synthetic short stories that use only a small vocabulary, generated by language models. The README links a paper that describes it.
The V2 training file is based only on generations from the stronger generator and is significantly larger. It contains all the examples in the original file that came from that generator.
Yes. The full archive includes a superset of the stories with metadata and the prompt used for each.
Yes. The README lists small models published on Hugging Face under roneneldan.