by Hugging Face
FineWeb-Edu is a 1.3 trillion token dataset of educational web pages filtered from Hugging Face's FineWeb using an educational-quality classifier. It is for teams pretraining or researching large language models.
DatasetFree
Step 1: Install the Hugging Face datasets library with pip install datasets.
Step 2: Pick a config: a sample such as sample-10BT, a single dump such as CC-MAIN-2024-10, or default for everything.
Step 3: Stream it with load_dataset("HuggingFaceFW/fineweb-edu", name="sample-10BT", split="train", streaming=True).
Step 4: For large-scale processing, read the Parquet files with datatrove's ParquetReader.
Step 5: Tokenize and feed the text into your pretraining pipeline.
This version has 1.3 trillion tokens. A larger version, FineWeb-Edu-score-2, has 5.4 trillion tokens.
ODC-By.
An open-weights language model scored 500k FineWeb samples for educational quality, a classifier was trained on those scores, and pages scoring at least 3 out of 5 were kept.
Yes. There are random samples of about 10B, 100B and 350B tokens (sample-10BT, sample-100BT, sample-350BT).
Hugging Face DatasetsHugging Face Datasets is the Hub's browsable index of machine-learning datasets, with full-text search and filters. It is for practitioners looking for datasets by task, modality, format or license.