by Hugging Face
FineWeb is Hugging Face's dataset of more than 18.5T tokens of cleaned and deduplicated English web text from Common Crawl. It is meant for pre-training language models.
DatasetFree
Step 1: Choose the full dataset, a single dump or a sample config such as the 10B-token sample.
Step 2: Load it with the datasets library, optionally in streaming mode.
Step 3: Or fetch specific files with huggingface_hub's snapshot_download and an allow pattern.
Step 4: Or read and filter documents with datatrove's ParquetReader.
Step 5: Use the table of dumps on the card to plan disk space.
The card says more than 18.5T tokens, up from the 15T tokens of the first release.
ODC-By 1.0.
Yes. There are configs for each Common Crawl dump and random samples of about 10B, 100B and 350B tokens.
English web data.
Hugging Face DatasetsHugging Face Datasets is the Hub's browsable index of machine-learning datasets, with full-text search and filters. It is for practitioners looking for datasets by task, modality, format or license.
MONETMONET is a dataset of 103.8 million English image-text pairs for training text-to-image models. It was distilled from 2.9 billion raw pairs through safety, domain and duplicate filtering.Humanity's Last ExamHumanity's Last Exam (HLE) is a benchmark of 2,500 expert-written questions across dozens of subjects, built to test AI models at the frontier of human knowledge. It is for researchers and model builders evaluating frontier AI.