by OpenBMB
Ultra-FineWeb is OpenBMB's filtered web pre-training dataset, with about 1T English tokens and 120B Chinese tokens. It is built by applying a verification-based quality filter to FineWeb and Chinese FineWeb data.
DatasetFree
Step 1: Open the dataset page on Hugging Face and read the card to choose between the English, Chinese, L1 and L3 variants.
Step 2: Check the license of each underlying source dataset for your intended use.
Step 3: Download or stream the files you need with the Hugging Face datasets library or hub tools.
Step 4: Optionally use the separate Ultra-FineWeb classifier to filter your own web text.
Step 5: Mix the data into your pre-training run and evaluate on your own benchmarks.
The card lists about 1T English tokens and about 120B Chinese tokens. The L3 release adds more than 400B English and 200B Chinese tokens.
The project is released under Apache 2.0. The card also says to check the license of each dataset it was built from.
L1 is the data after basic cleaning, heuristic filtering and deduplication. The main Ultra-FineWeb set is selected from it with the Ultra-FineWeb classifier.
Yes. The classifier is published as a separate Hugging Face repository and is built on fastText.