AI engineering connects a model to a useful workflow. You define a task, give the model only the context it needs, evaluate the result against examples, and design a safe path when it is unsure. This open course uses small design exercises so you can practise the decisions before paying for an API.
Define a narrow model task and a measurable success condition.
Separate retrieved evidence from generated wording.
Write an evaluation set with normal and failure cases.
Plan fallbacks for uncertain or unsupported answers.
[ READ · TRY · CHECK ]
Your lessons are ready.
Everything below is available without an account. Pause at each question and make a prediction before opening the answer.
01
LESSON 01 / 03
From demo to task
Turn a vague chatbot idea into a specific input, output, and acceptance check.
The idea
A useful AI feature has a bounded task and a visible success test.
“Build a smart tutor” is too broad to evaluate. A narrower task might be: answer a question using one approved lesson and show the line that supports the answer. Define the input, the allowed material, and what a valid response contains.
Write several example requests before choosing a model. Include one request the system should refuse or mark unknown. These examples become the first evaluation set.
WORKED EXAMPLE
Define a school FAQ task
Input: a student question. Output: a short answer, a source link, or “I could not find this in the approved guide.” Success: the answer is supported by the linked source.
YOUR TURN
Why is “sound helpful” a weak success criterion?
Need a hint?
Could two reviewers measure it the same way?
Check your answer
It is subjective and can reward confident errors. A checkable criterion names the evidence and the required response format.
Try it further: What would count as a correct answer to a school FAQ, and when should the system decline?
02
LESSON 02 / 03
Ground answers in evidence
Retrieve a relevant passage, keep its source, and make the answer traceable.
The idea
Retrieval can provide evidence, but generation must stay within it.
Split an approved document into sections with stable source links. Search for sections relevant to a question, pass only those sections to the model, and display the original source. The model can still misread or overstate a passage.
When two retrieved passages conflict, show that conflict or ask for clarification. Never turn an unsupported guess into a citation.
WORKED EXAMPLE
Resolve conflicting dates
One page says registration closes Monday and another says Wednesday. A grounded answer identifies both source dates and asks which schedule applies.
YOUR TURN
What should happen when search finds no relevant passage?
Need a hint?
Think about what the system actually knows.
Check your answer
State that the approved sources do not answer the question, and offer a way to inspect or contact the source owner.
Try it further: If two documents disagree, what should the answer show?
03
LESSON 03 / 03
Evaluate before shipping
Test ordinary, ambiguous, and out-of-scope requests with a small repeatable set.
The idea
Evaluation is a repeatable comparison between outputs and expected behaviour.
Build a small set of ordinary questions, tricky boundaries, ambiguous requests, and unanswerable questions. For each, record what evidence is allowed and what behaviour counts as success. Run the same set after every prompt, model, or retrieval change.
Track more than a single score: unsupported claims, missing citations, incorrect refusals, latency, and cost can move in different directions. Read failures, then add a case that would catch the mistake next time.
WORKED EXAMPLE
A five-case starter set
Two ordinary questions, one ambiguous question, one outdated source, and one question absent from all sources.
YOUR TURN
A new prompt answers more questions but invents two citations. Is it clearly better?
Need a hint?
Consider the cost of an unsupported answer.
Check your answer
No. Compare answer coverage and citation accuracy separately, then decide against the task’s stated requirements.
Try it further: Which failure cases would you include for a homework helper?
[ PUT IT TO WORK ]
Design a source-backed study helper
Sketch a helper that answers from a small approved reading set. Write the request format, a source display, five test cases, and a fallback response.
Check your work
Every answer points to an inspectable source.
The test set includes an unanswerable question.
The fallback clearly says what is unknown.
[ GO DEEPER / FOLLOW THE SOURCE ]
Further reading
These sources support further study. The lessons above are original AI Ka School material.