Ploba logo

Find the right AI tool for the job: apps, agents, APIs, MCP servers and datasets, explained plainly.

Get the AI Digest

Discover

  • Explore
  • Top Charts
  • Categories
  • Agents
  • MCP servers
  • Developer tools

Learn & community

  • Learn AI
  • AI Glossary
  • Blog
  • Hackathons
  • People
  • AI Digest

For makers

  • List your tool
  • Docs
  • Claim a listing
  • Advertise
  • Write for us

Company

  • About
  • Careers
  • Changelog
  • Trust
  • Security
  • Contact

© 2026 Ploba. All rights reserved.

  • Privacy
  • Terms
  • Cookies
  • Advertising
  • All policies
llms.txt
AI glossary
Generative AI

Token

Also called Tokenization

The basic unit of data—often a part of a word—that an AI model processes.

What is a Token?

In the context of Large Language Models, a token is the fundamental unit of data the model reads and generates. While humans read text word-by-word, AI models read text token-by-token. A token can be an entire word, a single syllable, or even just one character.

How does it work?

Before a model can process your prompt, the text runs through a "tokenizer," which chops the text into numerical IDs. As a general rule of thumb for English text, one token is roughly equivalent to 4 characters or 0.75 words. For example, the word "hamburger" might be split into three tokens: "ham", "bur", and "ger".

Why does it matter?

Tokens are the currency of generative AI. The limits of a model (its context window) are measured in tokens, not words. Furthermore, AI providers bill developers based on token usage. Understanding tokenization is crucial for optimizing cloud costs and predicting how much text a model can successfully read or write in a single interaction.

Category
Generative AI
Last reviewed
Sep 21, 2026
Also called
Tokenization

Sources

  • OpenAI: Tokenizer

Related terms

  • Generative AIContext WindowThe maximum amount of information, measured in tokens, that a model can process within one interaction.
  • ModelsLarge Language ModelA deep learning model trained on vast amounts of text to understand, generate, and predict human language.
  • Generative AITokenizerThe part of an AI system that splits text into tokens, the small pieces a language model reads and writes, and turns them into numbers.
  • DevelopmentInference CostWhat it costs to run an AI model on each request, usually priced per million input and output tokens through an API.