4.1.1. Building Test Sets

💡 First Principle: A test set is your agent's job description made executable: each test case pairs a realistic user input with the outcome that counts as success, so "the agent works" becomes a list of checkable claims instead of a feeling.

Composition is the craft. Good test sets cover the intent surface: every major topic and tool route, phrasing variety for the same intent (the orchestrator routes on meaning, so test meaning), edge cases (ambiguous requests, out-of-scope questions where the right behavior is declining or escalating), and the misconception-prone areas where wrong answers are costly. Test cases carry the user input plus whatever the chosen evaluation methods need — an expected answer for similarity- and match-based methods, expected keywords, or the expected tools/topics the agent should have used.

That last option deserves attention because it tests something answers can't: routing. A response can read beautifully while having come from general knowledge instead of the compliance index, or from the wrong tool entirely. Pairing an answer-quality method with a tool-use check catches the failure that looks like success.

Maintain test sets like the assets they are: grow them with every incident ("a user hit this — add it as a case"), and update expected outcomes when intended behavior changes, so the battery keeps measuring the agent you mean to have.

⚠️ Exam Trap: A test set that only checks answer text can pass while the agent routes wrong. When a scenario mentions verifying the agent used a particular tool, topic, or knowledge source, the answer involves the tool-use style of check — not another wording comparison.

Reflection Question: Draft the five test cases you'd write first for the refund agent from Phase 3 — and identify which one cannot be evaluated by comparing answer text at all.