A tokenizer is the tool that converts raw text into smaller pieces (tokens) and maps them to numbers the model can process. It's the component that runs tokenization, and it also detokenizes the model's output back into text.
In simple terms
The tool that cuts text into tokens and gives each a number — and turns tokens back into text.
How it works
- 1Raw text: you input text.
- 2Pre-tokenization: the text is cleaned and split into chunks.
- 3Subword tokenization: chunks are broken into subword tokens (e.g. via BPE merges).
- 4Vocabulary lookup: subwords are mapped to token IDs.
- 5Model input: the IDs are converted to embeddings for the model.
- 6Detokenize: the same tokenizer converts output tokens back into text.
Key points
- Common types: BPE (GPT, Llama), WordPiece (BERT), SentencePiece (Gemma, Mistral).
- Handles unknown words by breaking them into subwords.
- Efficient tokenization reduces text size while keeping meaning.
- Language-agnostic — works across many languages.
Why it matters
The tokenizer is the bridge between human text and model numbers. Its design affects how many tokens your text uses, which languages work well, and ultimately cost and speed.
Frequently asked questions
- What is BPE?
- Byte Pair Encoding — a method that merges frequent character pairs into subword tokens, used by GPT and Llama.
- Is the tokenizer part of the model?
- It's a paired component: each model ships with its own tokenizer, and both must match to encode and decode correctly.
More in Tokens & Embeddings
Token
The basic unit of text AI models read and generate.
Tokenization
Breaking text into tokens the model can read.
Token Embeddings
Turning tokens into meaningful vectors.
Embedding
Turning words and data into numbers AI understands.
Context Window
How much text an AI can consider at once.
Cosine Similarity
Measuring similarity by the angle between vectors.