5.4.1. Observability for Foundation Models and Agents
💡 First Principle: You cannot debug a model call you did not record. For FMs, the useful record is richer than a latency number: which model, how many input and output tokens, whether the call was throttled, and (when you choose to capture it) the prompt and the response themselves. Observability for generative AI is a question of which of these signals you have switched on.
Amazon Bedrock runtime metrics. Calls to the bedrock-runtime endpoint (Converse, ConverseStream, InvokeModel, InvokeModelWithResponseStream) publish metrics to CloudWatch in the AWS/Bedrock namespace, dimensioned by ModelId. These arrive without any setup:
| Metric | What it measures | Typical exam signal |
|---|---|---|
Invocations | Successful requests | "Request volume by model" |
InvocationLatency | Time from request sent to last token received | "Responses are slow end to end" |
TimeToFirstToken | Time to the first token, streaming APIs only | "Users wait before text starts appearing" |
InvocationClientErrors / InvocationServerErrors | Client-side and server-side failures | "Validation or permission errors" vs "service errors" |
InvocationThrottles | Throttled requests (these do not count as Invocations or errors) | "Requests rejected under load" |
InputTokenCount / OutputTokenCount | Tokens in and out | "Token consumption", "cost trend" |
CacheReadInputTokenCount / CacheWriteInputTokenCount | Prompt-cache reads and writes | "Is prompt caching actually hitting?" |
Two of these are worth separating carefully. InvocationLatency runs to the last token, so a response that is simply longer raises it even when the service is healthy. TimeToFirstToken isolates the wait before generation begins. If users complain that streaming responses "take a while to start", look at TimeToFirstToken; if they complain that complete answers take longer this week, check whether OutputTokenCount per request grew before blaming the service. AWS documents a metric-math check for exactly this on streaming traffic: output tokens per second, OTPS = OutputTokenCount / (InvocationLatency − TimeToFirstToken) × 1000. Latency rising with a stable OTPS means the responses got longer; a falling OTPS means the model is generating more slowly, which is the service-side signal worth alarming on.
Model invocation logging. Metrics are counts; logs are content. Model invocation logging records the full request, response and metadata for supported calls, including input and output token counts and the IAM identity that made the call. It is disabled by default, is configured per account and Region, and delivers to CloudWatch Logs, Amazon S3 or both (destinations must be in the same account and Region). Log events carry request and response bodies up to 100 KB; larger bodies and binary data such as images go to S3 only. Because a log record can contain the full prompt, treat the log group and bucket as sensitive data stores: encrypt them with a customer managed KMS key and apply CloudWatch Logs data protection where prompts may carry PII.
CloudWatch generative AI observability. CloudWatch now provides pre-built views for generative AI workloads instead of making you assemble dashboards by hand:
- Model Invocations — curated metrics on invocation count, token usage by model, latency, throttles and errors, plus an invocation table where you can open an individual request and read its input and output. The metrics views work from the runtime metrics above; the request-level table requires model invocation logging delivered to CloudWatch Logs.
- Amazon Bedrock AgentCore agents — session, latency, token and error metrics for agents and for the AgentCore Memory, Gateway, built-in tool and Identity resources they use, with end-to-end traces.
The views plug into the rest of CloudWatch, so the familiar tools from 5.2.1 still apply: you can alarm on any graph, send a request to Logs Insights, and add widgets to your own dashboards. Frameworks such as Strands Agents, LangChain and LangGraph can send traces to the same place.
Amazon Bedrock AgentCore Observability. For agents, AgentCore Observability emits telemetry in OpenTelemetry-compatible format and stores metrics, spans and logs in CloudWatch. Three setup facts decide most troubleshooting questions:
| Requirement | Why it matters |
|---|---|
| CloudWatch Transaction Search enabled once per account | Without it, the spans and traces that show each step of an agent run are not available; you see only the default metrics |
ADOT SDK (AWS Distro for OpenTelemetry) in the agent code, run with opentelemetry-instrument | Needed for the full trace detail and for custom metrics from your own code |
| Agents hosted outside AgentCore Runtime | Still supported: add the ADOT SDK (or, on Lambda, the ADOT Lambda layer) and set the OTEL environment variables. The standalone ADOT Collector is not supported for agent observability |
AgentCore Runtime publishes built-in metrics by default — invocations, throttles, system and user errors, latency, session count — and resource-usage metrics covered in 5.4.3. Gateway publishes per-tool invocation, error and duration metrics. Memory spans need tracing enabled on the memory resource, and built-in tools produce no logs by default.
Dashboards (4.2.3). A dashboard for an FM workload should put cost, speed and quality next to each other, because they trade against one another. A practical layout: token counts and cache hit ratio per model (cost), TimeToFirstToken and InvocationLatency p90 and p99 (speed), throttles and error counts (reliability), and guardrail interventions and evaluation scores (quality, from 5.4.2 and 5.4.7). CloudWatch dashboards can mix these sources; Amazon Quick can present the business-facing view.
⚠️ Exam Trap: "Enable CloudWatch metrics to see what the model said" is wrong. Runtime metrics never contain prompt or response text. Seeing the actual content of a request requires model invocation logging, and that is off until you turn it on.
⚠️ Exam Trap: An agent dashboard that shows invocation counts but no step-by-step traces usually means Transaction Search was never enabled (or tracing was never switched on for that resource), not that the agent lacks permissions. The default metrics arrive without it; the spans do not.
