A token is the basic unit of text that AI models read, process and generate. A token can be a whole word, part of a word, a number, punctuation or even a space — models see a sequence of tokens, not raw letters.
In simple terms
The small pieces of text a model actually reads — words, subwords, numbers or symbols.
How it works
- 1Text is split into tokens (e.g. 'unbelievable' → 'un' + 'believ' + 'able').
- 2Each token maps to a unique ID number.
- 3The model reads these IDs to understand the input.
- 4It generates output as tokens too, which are turned back into text.
Key points
- 1 token ≈ 3–4 characters, or about ¾ of a word in English.
- Context windows are measured in tokens (e.g. 8K, 128K).
- Cost is usually charged per token — input plus output.
- Different models tokenize the same text differently.
Why it matters
Tokens are the currency of LLMs: your context limit, speed and bill are all measured in them. Understanding tokens helps you write tighter prompts and predict cost.
Frequently asked questions
- Is a token the same as a word?
- Not always. Common words are one token, but longer or rare words split into several subword tokens.
- Why do tokens matter for cost?
- APIs charge by tokens (input + output), so fewer tokens mean lower cost and often faster responses.
More in Tokens & Embeddings
Tokenization
Breaking text into tokens the model can read.
Tokenizer
Breaks text into tokens the model can understand.
Token Embeddings
Turning tokens into meaningful vectors.
Embedding
Turning words and data into numbers AI understands.
Context Window
How much text an AI can consider at once.
Cosine Similarity
Measuring similarity by the angle between vectors.