3.6.8. Evaluating, Monitoring, and Auditing AI Workloads
💡 First Principle: An AI system can be healthy and wrong at the same time. Measure output quality with evaluations, operation with latency, token and error metrics, input change with drift detection, and keep a tamper-resistant record of what was asked and answered.
Scenario: A regulated lender must prove which prompts and answers its assistant produced, block a prompt or model change that lowers answer quality, alert when its agent starts failing tool calls, and detect when its SageMaker credit model sees data unlike its training set.
Evaluate before promoting (5.2.4). Amazon Bedrock Evaluations runs jobs against your prompt dataset:
- Programmatic (automatic metrics on built-in or custom datasets) and LLM-as-a-judge (built-in metrics such as correctness, completeness, faithfulness, helpfulness, relevance, harmfulness and refusal, or custom metrics).
- Human-based jobs with your own reviewers for subjective quality.
- RAG evaluations, retrieve-only or retrieve-and-generate, for a knowledge base or an external RAG system.
- Bring your own responses to evaluate models or RAG pipelines that run outside Bedrock.
Put an evaluation step in the deployment pipeline: when a prompt template, model version or knowledge base configuration changes, run the job on a golden dataset and fail the promotion if scores drop below thresholds. AgentCore Evaluations scores agent sessions, traces and spans with built-in or custom evaluators, on demand or online against live traffic.
Monitor in production (5.3.5).
- Model and inference: Bedrock publishes
Invocations,InvocationLatency,TimeToFirstToken,InputTokenCount,OutputTokenCount,InvocationThrottlesand error counts to CloudWatch (namespaceAWS/Bedrock, dimensionModelId); guardrails publish intervention metrics. SageMaker AI endpoints publishModelLatency,OverheadLatency, invocations, errors and utilization. - Agents: AgentCore Observability emits OpenTelemetry spans for every reasoning step and tool call, plus session count, latency, duration, token usage and error rates, into CloudWatch, with a generative AI observability view for agent traces (enable CloudWatch Transaction Search first).
- Drift: SageMaker Model Monitor is closed to new customers, so a new design enables Data Capture on the endpoint (requests and predictions to S3), runs a scheduled drift analysis (EventBridge with Lambda or a SageMaker Pipeline, using open-source tooling such as Evidently), publishes the drift score as a CloudWatch custom metric with an alarm, and joins predictions with delayed ground-truth labels by inference ID to track model accuracy.
Audit trail (2.3.2, 2.4.3). CloudTrail records InvokeModel and Converse as management events (who called which model, when, and in which Region inference ran) but not the prompt or response. Retrieve, RetrieveAndGenerate, InvokeAgent and ApplyGuardrail are data events that need advanced event selectors. Model invocation logging captures the full request, response and metadata, including token counts and caller ARN. It is disabled by default and configured per account and per Region, and its CloudWatch Logs or S3 destination must be in the same account and Region. For an organization, deploy the logging configuration with StackSets and centralize the logs into the log archive account (S3 replication or CloudWatch Logs subscriptions), encrypted with KMS, retained to policy and protected with S3 Object Lock where tamper resistance is required. Guardrail-blocked content appears in these logs in plain text, so restrict who can read them.
⚠️ Exam Trap: An organization CloudTrail trail is not a record of what the model said. When a requirement is to reproduce prompts and responses for auditors, the answer must include model invocation logging, enabled before the conversations happen.