2.5.1. Evaluators: Groundedness, Relevance, Coherence, Safety

💡 First Principle: Each evaluator answers a specific question about output quality, and choosing the right evaluator means knowing which question matters for the requirement. Groundedness asks "is it supported by the provided source?"; relevance asks "does it address the query?"; coherence/fluency ask "is it well-formed?"; safety evaluators ask "is it harmful?"; agent evaluators (tool-call accuracy, intent resolution, task adherence) ask "did the agent work correctly?"

The Azure AI Evaluation SDK provides these as built-in, AI-assisted evaluators (an LLM judge scores each output, often pass/fail). For a RAG system, groundedness is the headline metric — it catches the model answering beyond its sources. For agents, tool-call accuracy and task adherence catch the agent choosing wrong tools or drifting off-task. Safety/red-team scans probe for harmful generations.

⚠️ Exam Trap: Don't reach for a lexical-overlap metric (like F1/exact match) to score a RAG answer — a correct but paraphrased answer scores poorly on word overlap. Semantic evaluators (groundedness, relevance) are the right tool, which is exactly why Foundry de-emphasizes lexical metrics for these workflows.

Reflection Question: A RAG assistant gives fluent, confident answers that QA loves — but some aren't actually in the source documents. Which single evaluator surfaces this problem, and why won't relevance or coherence catch it?