Tokenizer

What is a tokenizer?

A tokenizer is the component that turns text into tokens before a large language model sees it, and turns tokens back into text afterwards. A token might be a whole common word ("the"), part of a word ("tok" + "enizer"), a space, a punctuation mark or a single character. Each token has an ID number, and those numbers are what the model actually works with.

How does it work?

Most modern tokenizers use a method such as byte-pair encoding. Before training, the tokenizer studies a large amount of text and builds a fixed vocabulary, often tens of thousands of tokens or more: frequent words get a single token, while rare words are built from several pieces. Every model family has its own tokenizer, and the model can only read text split by that tokenizer.

What is it commonly confused with?

• Words. Tokens aren't words. In English, a rough guide is about four characters, or three-quarters of a word, per token, but this varies by model and content.
• Chunking. Chunking splits documents into passages for search. Tokenizing splits text into the tiny units a model reads.

Why does it matter?

The tokenizer decides how many tokens your text becomes, and tokens decide what you pay (inference cost) and how much fits in the context window. The same text can be a different number of tokens in different models. Languages other than English, code and numbers often need more tokens per word, so they cost more and fill the window faster. Tokenization also explains some odd mistakes, such as models miscounting letters in a word they see as just one or two tokens.