Inference engineering is the work of serving a trained model to real users within latency, cost, and reliability limits. This course builds a mental model of prefill and decoding, explains why the KV cache grows, and shows how to benchmark useful outcomes rather than a single speed number.
Distinguish prefill time from token-by-token decoding.
Explain what a KV cache stores and why memory matters.
Compare batching and quantization trade-offs.
Choose useful latency and throughput measurements.
[ READ · TRY · CHECK ]
Your lessons are ready.
Everything below is available without an account. Pause at each question and make a prediction before opening the answer.
01
LESSON 01 / 03
Follow one request
Trace prompt tokenisation, prefill, first token, decoding, and final response.
The idea
Prefill and decoding put different pressure on a model server.
A language model first processes the input prompt to prepare attention state; this is prefill. It then generates output tokens one by one; this is decoding. A long input can increase time to first token, while a long output extends the decoding phase.
Measure both phases. “The request took three seconds” hides whether users waited before anything appeared or watched a slow stream.
WORKED EXAMPLE
Compare two requests
A long document with a two-word answer stresses prefill; a short prompt asking for a long essay stresses decoding.
YOUR TURN
Why can two requests with the same output length have different time to first token?
Need a hint?
Check how much input each request must process.
Check your answer
Their prompt lengths, batching conditions, or cache states may differ, changing prefill time.
Try it further: Why might two requests with the same output length have different time to first token?
02
LESSON 02 / 03
Memory and batching
Reason about the KV cache, concurrent requests, and how batches share compute.
The idea
The KV cache trades memory for avoiding repeated attention work.
During decoding, attention needs keys and values from earlier tokens. Keeping them in memory avoids recomputing the entire history for each new token. Longer contexts and more active sequences therefore consume more cache memory.
Batching can improve hardware use but raises scheduling and memory questions. Quantization reduces precision and can reduce memory use, but quality and hardware speed must be measured rather than assumed.
WORKED EXAMPLE
Plan concurrency
If each active request keeps a growing cache, the number and length of concurrent conversations constrain how many can run together.
YOUR TURN
What changes when many long conversations are active?
Need a hint?
Each conversation retains past-token state.
Check your answer
KV cache memory rises, potentially limiting concurrency or requiring eviction, shorter contexts, or different scheduling.
Try it further: What happens to memory when many long conversations are active?
03
LESSON 03 / 03
Measure the service
Compare time to first token, tokens per second, tail latency, and quality under load.
The idea
A benchmark should describe real traffic and the model setting.
Record time to first token, output tokens per second, complete response latency, and tail latency under several concurrency levels. Use short and long prompts, varied output lengths, and a quality check. Publish model version, hardware, precision, and batch settings.
A single unloaded run may look fast while users still experience queuing at peak traffic. Plot latency against throughput so the trade-off is visible.
WORKED EXAMPLE
Build a test matrix
Measure 200-token and 4,000-token prompts at 1, 8, and 32 concurrent requests, with short and long responses.
YOUR TURN
Why might a faster tokens-per-second number still feel worse to users?
Need a hint?
Consider waiting before the first token and requests at the slow end.
Check your answer
Queuing or slow prefill can increase time to first token or tail latency even if decoding throughput is high.
Try it further: Why is a single unloaded benchmark a poor estimate of user experience?
[ PUT IT TO WORK ]
Write a serving benchmark plan
Specify short and long prompts, concurrency levels, output lengths, quality checks, and what you would chart.
Check your work
The plan measures both latency and throughput.
It includes a long-context case.
It records model and hardware settings for fair comparison.
[ GO DEEPER / FOLLOW THE SOURCE ]
Further reading
These sources support further study. The lessons above are original AI Ka School material.