4.1.2. Evaluation Methods and Reviewing Results
💡 First Principle: The evaluation method must match the variability of a correct answer. Fixed answers tolerate zero variation — match exactly. Meaning-fixed answers vary in wording — compare semantics. Open-ended answers vary in everything — judge quality. Mismatch method to variability and you manufacture false failures or false confidence.
Copilot Studio ships seven methods; the exam-ready map:
| Method | Scoring | Choose when |
|---|---|---|
| Exact Match | Pass/Fail | One right string — codes, numbers, fixed phrases |
| Keyword Match | Pass/Fail (any/all) | Specific terms must appear (or a required disclaimer) |
| Text Similarity | 0–100% vs. threshold | Wording and structure should closely track the expected text |
| Compare Meaning | 0–100% vs. threshold | Many phrasings, one meaning — the workhorse for factual answers |
| General Quality | 0–100% | No expected answer exists — judge relevance, groundedness, completeness, abstention |
| Tool Use | Pass/Fail | The route matters — verify specific tools/topics were used |
| Custom | Pass/Fail by label | Your own criteria and instructions — compliance rules, policy language |
Methods combine per test case — Compare Meaning for the content plus Tool Use for the route plus Keyword Match for the mandatory disclaimer is a single case's rubric, not three cases.
Reviewing results is diagnosis, not scorekeeping: read failures for pattern — routing misses cluster around description problems; content misses cluster around knowledge and instructions; and some failures indict the test case itself (stale expected answers, missing method configuration showing up as invalid results). Fix, re-run, and keep the passing run as the baseline the next change must beat.
⚠️ Exam Trap: "Responses are correct but phrased differently each run, and evaluations keep failing" — the method is too strict for the answer's natural variability. Step down the ladder: Exact Match → Text Similarity → Compare Meaning → General Quality until strictness matches reality.
Reflection Question: Assign methods: (a) the agent must return a part number, (b) must explain a policy in any wording, (c) must always include a legal disclaimer, (d) must have consulted the HR specialist agent. Defend each pick in one clause.
