Many AI projects are judged by a few demos. Someone tries five questions, the answers look good, and the team moves on. Then real users arrive with questions nobody tried, and quality turns out to be much lower than expected. Later, a small change to a prompt or model quietly makes things worse, and nobody notices.
The fix is simple and does not require data scientists: a test set, sometimes called an evaluation set. It is a list of real examples with a clear idea of what a good result looks like. Here is how to build one.
What a test set is
A test set is a list of inputs paired with what you expect:
- For a support assistant: real customer questions, the correct answer, and the document it should come from.
- For document extraction: real invoices and the correct values for each field.
- For classification: real messages and the correct category.
A spreadsheet is enough to start. You run the examples through the AI, compare the results with what you expected, and record a score.
Step 1: Collect real examples
Use real data wherever you can, with personal details removed. Thirty to fifty examples is a useful starting point for a pilot; you can grow it over time.
Aim for a realistic mix:
- Common cases that make up most of the volume
- Hard cases, such as vague wording, typos, or several questions at once
- Out-of-scope cases, where the right response is "I don't know" or a handoff
- Risky cases, where a wrong answer would be costly
Step 2: Define what "good" means
Write down the criteria before you look at the results, so you are not tempted to adjust them later. For an assistant, you might check:
- Is the answer correct?
- Does it cite the right source?
- Does it decline when it should?
- Is the tone acceptable?
Keep the scoring simple, such as pass or fail for each criterion. Ask a person who knows the subject to do the judging, ideally two people for a sample, to check that you agree on what good looks like.
Step 3: Run it and read the failures
Run the full set and calculate the pass rate, but spend most of your time reading the failures. They usually fall into a few groups:
- Missing or outdated content
- The wrong document retrieved
- Unclear instructions to the model
- Questions that should have been handed to a person
Each group points to a specific fix, and content fixes are often the cheapest.
Step 4: Decide what is good enough
There is no universal target. The bar depends on what a mistake costs. A tool that drafts replies for a person to review can launch with a lower pass rate than one that answers customers directly. Agree on the threshold before launch, and set a stricter one for risky cases, which may need to pass every time.
Step 5: Use it every time something changes
This is where the test set really pays off. Run it whenever you change a prompt, a model, or your content, and compare the score with the previous run. It turns "this feels better" into evidence, and catches regressions before users do.
Keep it alive after launch
- Add real failures from production to the test set.
- Review a sample of live conversations regularly.
- Watch signals from users, such as thumbs down, handoffs, and repeated questions.
- Remove examples that no longer apply, and update expected answers when policies change.
Example: a returns assistant
A store building a returns assistant could collect 40 real questions, including a few it should refuse, and record the correct policy article for each. The first run might show that most failures come from an outdated article. After fixing it, the team reruns the set and compares the results before deciding whether to widen the pilot. This is an illustrative example; your own thresholds depend on your risks.
The bottom line
You cannot improve what you do not measure. A small set of real examples, clear criteria, and a habit of rerunning it will tell you more than any demo.
If you want help building a test set or evaluating an AI feature before launch, see our AI agents and knowledge assistant service or email us.
