What is a Transformer?

The Transformer is a deep learning architecture that powers most modern AI models, including GPT, Claude, Gemini and Llama. Its key idea is attention: the model looks at all words in a sequence at once and decides which ones matter most for each prediction.

How it works

1. 1The input text is split into tokens and given positions.
2. 2The encoder reads all tokens and builds rich representations.
3. 3The decoder generates output tokens one by one.
4. 4Self-attention helps the model focus on the most relevant words.
5. 5The model is trained to predict the next token in a sequence.

Key points

• Attention = focus on what matters most, across the whole input at once.
• Unlike older RNNs, Transformers handle long-range context and train in parallel, so they scale well.
• An encoder stack reads; a decoder stack generates — GPT-style models use the decoder.
• It works across text, code, images and audio.

Why it matters

The Transformer is the reason today's language models are so capable. Attention lets a model weigh every word against every other word, capturing meaning and context far better than the sequential models that came before.

Frequently asked questions

What is self-attention?

A mechanism that lets each token look at all other tokens and decide which are most relevant when producing the next output.

Why did Transformers replace RNNs?

They understand long-range context better and process a whole sequence in parallel, making them faster to train and far more scalable.