What is a Token?
In the context of Large Language Models, a token is the fundamental unit of data the model reads and generates. While humans read text word-by-word, AI models read text token-by-token. A token can be an entire word, a single syllable, or even just one character.
How does it work?
Before a model can process your prompt, the text runs through a "tokenizer," which chops the text into numerical IDs. As a general rule of thumb for English text, one token is roughly equivalent to 4 characters or 0.75 words. For example, the word "hamburger" might be split into three tokens: "ham", "bur", and "ger".
Why does it matter?
Tokens are the currency of generative AI. The limits of a model (its context window) are measured in tokens, not words. Furthermore, AI providers bill developers based on token usage. Understanding tokenization is crucial for optimizing cloud costs and predicting how much text a model can successfully read or write in a single interaction.