Ploba logo

Find the right AI tool for the job: apps, agents, APIs, MCP servers and datasets, explained plainly.

Get the AI Digest

Discover

  • Explore
  • Top Charts
  • Categories
  • Agents
  • MCP servers
  • Developer tools

Learn & community

  • Learn AI
  • AI Glossary
  • Blog
  • Hackathons
  • People
  • AI Digest

For makers

  • List your tool
  • Docs
  • Claim a listing
  • Advertise
  • Write for us

Company

  • About
  • Careers
  • Changelog
  • Trust
  • Security
  • Contact

© 2026 Ploba. All rights reserved.

  • Privacy
  • Terms
  • Cookies
  • Advertising
  • All policies
llms.txt
Ploba logo
DiscoverHackathonsPeopleLearnAI Digest
Sign inGet started
HomeTop ChartsExploreCategories
TypesWeb apps1,894Agents585Plugins416APIs376Mobile apps152Datasets85
AdvertisingYour product hereReach people browsing AI tools on Ploba.Advertise hereAdvertisingYour product hereReach people browsing AI tools on Ploba.Advertise here
  1. Explore
  2. /
  3. Developer Tools
  4. /
  5. Common Crawl
Developer Tools
Common Crawl icon

Common Crawl

by Common Crawl

Common Crawl is a free, open repository of web crawl data maintained by a non-profit. Its archives are published roughly monthly and used for research and language model training.

Dataset·Free

Visit commoncrawl.org(opens in a new tab)

About Common Crawl

Key features of Common Crawl

  • Raw crawl data in WARC format, including full HTTP responses
  • WAT files with metadata, HTTP headers and link graphs
  • WET files with plain text extracted from HTML, suited to NLP work
  • CDXJ and URL indexes for each monthly release, queryable with Amazon Athena
  • Host-level and domain-level web graphs with Harmonic Centrality and PageRank values
  • Bulk download from S3 or direct access for cloud processing
  • Opt-out registry and robots.txt support for site owners

Who Common Crawl is for

  • Researchers studying the web
  • Machine learning teams building language model training data
  • NLP and language identification researchers
  • Companies and individuals analyzing web-scale data

How to use Common Crawl

  1. 1

    Step 1: Browse the latest crawl and its stats on the Common Crawl site.

  2. 2

    Step 2: Pick a format: WARC for raw responses, WAT for metadata, or WET for extracted text.

  3. 3

    Step 3: Download files from S3 in bulk, or use the cc-downloader client.

  4. 4

    Step 4: Use the URL Index with Amazon Athena or Apache Spark to filter large crawls.

  5. 5

    Step 5: Review the terms of use before building on the data.

  6. 6

    Step 6: Keep request rates low when calling the CDX index API, and use HTTPS.

Common Crawl FAQ

Is Common Crawl free?

Yes. Common Crawl is a non-profit and provides its archives and datasets to the public at no cost.

How can I stop CCBot from crawling my site?

Add a Disallow rule for CCBot to your robots.txt file. You can also ask to be added to the opt-out registry.

Does Common Crawl archive entire websites?

Not generally. The dataset is a sample of the web, with a randomly selected subset of pages from each site.

What formats is the data available in?

Raw data is in WARC format, with WAT files for metadata and link graphs and WET files for extracted plain text.

Details

Pricing
FreeFree and open. Common Crawl applies as few restrictions as possible, with terms of use meant to prevent abusive or illegal usage. The CDX index API is rate limited.
Type
Dataset
Categories
Developer ToolsResearch
Added
Sep 15, 2026
Common Crawl iconPublisherCommon Crawl

Built from public sources

We put this page together from Common Crawl’s own website and docs so you can compare it with other tools. Ploba isn’t affiliated with Common Crawl.

Work at Common Crawl? Claim the listing to manage it yourself.

Something wrong here?

Related tools

See more
PhysioNet iconPhysioNetPhysioNet · FreePhysioNet is an NIH-funded platform run by the MIT Laboratory for Computational Physiology that hosts curated, de-identified biomedical and clinical datasets, software and models. Most resources are openly available; some require credentialing.Developer Tools · FreeLAION iconLAIONLAION · FreeLAION is a non-profit research network that publishes open datasets, models and tools for machine-learning research. Its datasets are indexes of links and captions, not stored media.Developer Tools · FreeKaggle iconKaggleKaggle · FreeKaggle is a Google-owned data science platform where users find and publish datasets, run code in a browser-based environment and enter machine learning competitions.Developer Tools · Free