3.4.1. Observability: Tracing, Metrics, and Continuous Evaluation
💡 First Principle: Three observability capabilities work together — metrics (Azure Monitor / Application Insights dashboards for token use, latency, error rate, quality scores), tracing (OpenTelemetry spans capturing each step and tool call within a run), and continuous evaluation (running the Phase 2 evaluators against production traffic to catch drift). Each answers a different operational question.
Foundry integrates with Azure Monitor Application Insights for real-time dashboards tracking operational metrics and quality scores, with alerting when outputs fail quality thresholds or produce harmful content. Tracing via OpenTelemetry captures the detailed execution flow — essential for debugging why an agent took a path or which tool call failed inside a multi-step run. Continuous evaluation applies groundedness/relevance/safety evaluators to live traffic so quality regressions surface early rather than via user complaints.
⚠️ Exam Trap: When a scenario asks how to debug why an agent made a particular sequence of tool calls, the answer is tracing (OpenTelemetry spans), not a metrics dashboard. Metrics tell you that latency rose; traces tell you what happened inside the run. Picking the dashboard for a step-level debugging requirement is the wrong-granularity trap.
Reflection Question: Latency dashboards look fine, but users report the agent "sometimes does the wrong thing." Which observability capability surfaces the cause, and why can't the metrics dashboard alone explain it?
