Tokenization is the process of breaking text into smaller pieces called tokens, which are then converted into numbers a model can understand. It is the first step in how an AI reads your input.
In simple terms
Turning raw text into the numbered tokens a model can actually process.
How it works
- 1Input text: you provide some text.
- 2Tokenizer: it splits the text into tokens.
- 3Token IDs: each token is mapped to a unique ID.
- 4Model input: these IDs are fed to the model.
- 5Model output: generated tokens are converted back into text.
Key points
- Token types include whole words, subwords, punctuation and special tokens.
- Token IDs vary by model and tokenizer (GPT, Llama, Claude differ).
- Tokenization affects cost, context usage and performance.
- It's the foundation of all NLP and LLM applications.
Why it matters
Tokenization decides how efficiently text is represented. Good tokenization keeps meaning intact while using fewer tokens, which improves both cost and how much you can fit in the context window.
Frequently asked questions
- Does every model tokenize the same way?
- No. Each model has its own tokenizer, so the same sentence can be a different number of tokens across models.
- Why not just use characters or words?
- Subword tokens balance vocabulary size and flexibility, handling new or rare words gracefully.
More in Tokens & Embeddings
Token
The basic unit of text AI models read and generate.
Tokenizer
Breaks text into tokens the model can understand.
Token Embeddings
Turning tokens into meaningful vectors.
Embedding
Turning words and data into numbers AI understands.
Context Window
How much text an AI can consider at once.
Cosine Similarity
Measuring similarity by the angle between vectors.