5.4.3. Monitoring Agent Performance, Coordination and Resource Use
💡 First Principle: An agent is a loop: the model decides, a tool acts, the model reads the result and decides again. Most agent failures are not model errors in the HTTP sense. The tool times out, returns an error the model paraphrases away, or the loop keeps going until it runs out of budget. Monitoring an agent means instrumenting each step of the loop, not only the request that started it.
Tool failures. Tool calls fail silently from the user's point of view: the model receives an error payload and often writes a polite apology, so the overall request still returns success. The evidence lives in the step records:
- Amazon Bedrock Agents Classic traces (existing agents only;
enableTrace) include anOrchestrationTracewith the action group input (which API or function, which parameters) and the observation (what the action group returned), and aFailureTracewith the reason a step failed. - AgentCore Gateway publishes per-tool metrics in the
AWS/Bedrock-AgentCorenamespace, with the tool name as a dimension:Invocations,Throttles,SystemErrors(5xx),UserErrors(4xx other than 429),Latency(to first response token),Duration(end to end) andTargetExecutionTime(time spent in the Lambda, OpenAPI or MCP target). HighTargetExecutionTimewith normal gateway overhead points at the tool itself, not the gateway. These gateway metrics are found under All metrics in CloudWatch; AWS notes they are not shown on the generative AI observability page. - AgentCore Runtime records user, system and throttling error types on its spans, so you can separate a malformed request from a service fault from a quota problem.
Truncated streaming responses. A streamed answer that stops mid-sentence has three common causes, and the response tells you which:
| Evidence | Meaning | Fix |
|---|---|---|
stopReason = max_tokens in messageStop | Hit the output-token limit you set | Raise maxTokens, or ask for shorter output |
stopReason = model_context_window_exceeded | Prompt plus history filled the context window | Trim or summarise history, reduce retrieved context |
stopReason = guardrail_intervened or content_filtered | A safeguard stopped the output | Review guardrail trace (5.4.7) |
A modelStreamErrorException, throttlingException or internalServerException event mid-stream, and no messageStop | The stream broke | Retry with backoff; alarm on the error rate |
Normal completions end with end_turn (or stop_sequence when you configured stop sequences); an agent loop pausing to call a tool ends with tool_use. Treat any other value as something to count. An application that logs stopReason per request can emit a custom metric such as "truncated responses per minute" and alarm on it. For SageMaker AI endpoints that stream responses, the endpoint-level equivalent is the MidStreamErrors metric.
Coordination failures. In a multi-agent system, a supervisor routes work to collaborators. Coordination fails when the supervisor picks the wrong collaborator, collaborators hand work back and forth, or one collaborator's output is not in the shape the next one expects. AgentCore traces show the whole span tree for a session, and on existing Agents Classic multi-agent setups the traces carry a RoutingClassifierTrace and a callerChain showing which agents forwarded the request. Automate the response: alarm when the number of steps or model calls per session exceeds a ceiling (a loop signature), and set iteration limits in the orchestration code so a stuck loop ends instead of spending indefinitely.
Automating quality checks on agents. Amazon Bedrock AgentCore Evaluations scores agent sessions with LLM-as-a-judge evaluators — built-in ones such as Builtin.Helpfulness, or custom LLM or Lambda code-based evaluators. Online evaluation samples a percentage of live sessions (for example 10%) and tracks the scores over time; on-demand evaluation scores specific traces you pick; batch evaluation scores a set of sessions from CloudWatch Logs, which suits before-and-after comparisons when a prompt or model changes.
Agent resource consumption (4.2.8). AgentCore Runtime bills CPU and memory per second of actual consumption, and CPU is not charged during I/O wait (for example, while the agent waits for a model response or a tool call) when no other background process is running. It publishes CPUUsed-vCPUHours and MemoryUsed-GBHours at account, agent and endpoint level, with session-level usage logs (USAGE_LOGS) at one-second granularity. Watch these for:
- Runaway sessions — a session whose memory-hours keep climbing after the user has gone.
- Heavy background work — CPU hours that do not fall during waits suggest processing that is not I/O-bound.
- Session growth —
SessionCountcounts new sessions;ActiveSessionCountis a live gauge that also shows how close you are to session quotas.
Memory is billed on the peak memory consumed up to each second, with a 128 MB minimum, and resource-usage data can arrive up to 60 minutes late. The telemetry is for monitoring; the bill is the authority, so reconcile against Cost Explorer rather than treating the metrics as invoice values.
⚠️ Exam Trap: "The 5xx error rate is zero, so the agent's tools are healthy" is a trap. A failing tool usually produces a successful agent response that apologises or guesses. Look at the trace observations, the gateway's per-tool error metrics, or an evaluation score — not the top-level error rate.
⚠️ Exam Trap: A response that ends mid-sentence with stopReason = max_tokens is a configuration issue, not a throttling issue. Retrying the same request with the same maxTokens will truncate again.
