by OpenBMB
UltraData-Code is OpenBMB's code dataset for language model pretraining, released at two tiers: about 400B tokens of algorithmically relevant code and about 150B tokens of synthesized programming exercises. Both cover 11 programming languages.
DatasetFree
Step 1: Open the dataset page on Hugging Face and read the card.
Step 2: Choose the L2 tier for selected source files or the L3 tier for programming exercises.
Step 3: Filter L2 rows by category, algorithmic relevance score or quality score to fit your training goal.
Step 4: Mix L2 and L3 data in your continual pretraining or training run.
Step 5: Evaluate the resulting model on benchmarks such as EvalPlus and MultiPL-E.
This release has two tiers: L2 with about 400B tokens of algorithmically relevant code and L3 with about 150B tokens of synthesized programming exercises.
Both tiers cover 11 programming languages.
Each row is one source file with a uuid, repository name, relative path, content, predicted category, an algorithmic relevance score from 0 to 1 and a quality score from 0 to 10.
The pipeline starts from approximately 192 million public GitHub repositories archived at L0.