
by Common Crawl
Common Crawl is a free, open repository of web crawl data maintained by a non-profit. Its archives are published roughly monthly and used for research and language model training.
DatasetFree
Step 1: Browse the latest crawl and its stats on the Common Crawl site.
Step 2: Pick a format: WARC for raw responses, WAT for metadata, or WET for extracted text.
Step 3: Download files from S3 in bulk, or use the cc-downloader client.
Step 4: Use the URL Index with Amazon Athena or Apache Spark to filter large crawls.
Step 5: Review the terms of use before building on the data.
Step 6: Keep request rates low when calling the CDX index API, and use HTTPS.
Yes. Common Crawl is a non-profit and provides its archives and datasets to the public at no cost.
Add a Disallow rule for CCBot to your robots.txt file. You can also ask to be added to the opt-out registry.
Not generally. The dataset is a sample of the web, with a randomly selected subset of pages from each site.
Raw data is in WARC format, with WAT files for metadata and link graphs and WET files for extracted plain text.
KaggleKaggle is a Google-owned data science platform where users find and publish datasets, run code in a browser-based environment and enter machine learning competitions.