Token Embeddings

Token embeddings turn tokens into meaningful high-dimensional vectors so a model can understand their meaning and relationships. Similar tokens end up with similar vectors, encoding meaning as geometry.

How it works

1. 1Input tokens: text is split into tokens.
2. 2Lookup: each token is looked up in the embedding matrix.
3. 3Vector representation: each token becomes a high-dimensional vector.
4. 4Used by model: these vectors are fed to the model for prediction.
5. 5Capture meaning: similar tokens have similar vectors, so relationships are captured.
6. 6Better understanding: the model understands context and meaning more effectively.

Key points

• Tokens become dense vectors (e.g. 768 dimensions).
• Relationships like king − man + woman ≈ queen are captured as geometry.
• Embeddings are learned during model training.
• They are the first layer that helps a model understand language.

Why it matters

Embeddings are how models represent meaning numerically. They let models compare, cluster and reason over tokens, and they power semantic search, RAG and recommendations.

Frequently asked questions

How are token embeddings different from one-hot encoding?

One-hot is sparse and meaningless; embeddings are dense and capture semantic similarity between tokens.

Are embeddings learned or fixed?

They are learned during training, so the model discovers which tokens are related.