Inference is the process of using a trained AI model to make predictions or generate outputs on new, unseen data. Training is where a model learns patterns; inference is where it puts them to work in real applications.
In simple terms
Training is learning the patterns; inference is using them to make decisions.
How it works
- 1Input: new, unseen data is provided to the model.
- 2Preprocess: the input is tokenized/normalised into a format the model understands.
- 3Model (forward pass): the input passes through the model to produce probabilities or values.
- 4Generate: the model produces the most likely tokens or output.
- 5Decode & output: results are converted back into the final prediction or text.
Key points
- During inference the model's weights are fixed — no learning happens.
- It is optimised for speed and cost using quantization, caching and batching.
- Runs everywhere: cloud GPUs, edge devices and phones.
- Cost usually depends on usage — more tokens or larger models cost more.
Why it matters
Inference is where an AI model earns its keep — every chatbot reply, search result or recommendation is an inference. Its speed and cost decide whether an AI feature is actually usable in production.
Frequently asked questions
- Does the model learn during inference?
- No. The weights are frozen; it only applies what it already learned during training.
- Why is inference cost important?
- You pay for every request. Optimising inference (smaller models, caching, batching) is often the biggest lever on an AI product's economics.
More in AI Foundations
What is AI (Artificial Intelligence)?
How AI works and its types — narrow, general, super.
What is Machine Learning?
Supervised, unsupervised and reinforcement learning.
What is Deep Learning?
Multi-layer neural networks that learn features.
What is a Neural Network?
Layers of connected neurons that learn patterns.
What is a Transformer?
The attention-based architecture behind modern AI.
Batch Inference
Process many inputs at once for efficiency.