2.1.5. Monitoring AI and Container Workloads: Amazon Bedrock and Amazon Managed Service for Prometheus

💡 First Principle: A generative AI call is an API call with three extra dimensions: a cost measured in tokens, a quota measured in tokens per minute, and a payload (the prompt and the response) that is exactly what you need when an answer goes wrong. Monitoring an AI workload means watching all three, and each one lives in a different place.

Exam guide v1.1 (June 2026) adds AI workloads to skill 1.1.1 next to serverless and compute, and names Amazon Managed Service for Prometheus alongside CloudWatch and CloudTrail. The operational question is always the same: which tool holds the signal I need, and is it switched on?

The Three Bedrock Signals:

Bedrock Runtime Metrics Worth Alarming On:

Throttled requests are counted neither as Invocations nor as errors, and how many throttles you see depends on your SDK's retry settings. A client that retries aggressively can hide a quota problem.

Why Bedrock Throttles "Early": When a request starts, Bedrock deducts input tokens + max_tokens from your tokens-per-minute (TPM) quota. It adjusts the charge during processing and gives back the unused reservation at the end. Some models also apply an output-token burndown rate (for example, one output token consuming five tokens of quota). An application that sets max_tokens far above its real reply size uses up its quota in reservations while its actual usage looks modest. Fix it in this order:

1. Compare OutputTokenCount with the max_tokens the application sends, and right-size max_tokens.
2. If actual consumption is genuinely near the quota, request a quota increase.
3. For sustained high throughput on a model, consider Provisioned Throughput.

Model Invocation Logging: This is the only Bedrock source that holds the content: the full request, the full response, and metadata including identity.arn and the input and output token counts. Key operating rules:

• Disabled by default. Enable it in the Bedrock console under Settings, or with PutModelInvocationLoggingConfiguration. It applies to the account in one Region.
• Destinations: CloudWatch Logs, Amazon S3, or both, and they must be in the same account and Region as the logging configuration. To centralize logs in another account, deliver locally and replicate or forward from there.
• Size split: input and output JSON bodies up to 100 KB go inline. Larger bodies and binary image output go only to Amazon S3, so a CloudWatch-Logs-only configuration also needs an S3 location for large data.
• Delivery health: Bedrock emits ModelInvocationLogsCloudWatchDeliveryFailure, ModelInvocationLogsS3DeliveryFailure and ModelInvocationLargeDataS3DeliveryFailure (with matching …Success metrics) in AWS/Bedrock. Alarm on the failures.
• Scope: calls through the bedrock-runtime endpoint. Calls through the separate bedrock-mantle endpoint are not captured by invocation logging.

Because every record carries the caller's ARN and token counts, Logs Insights can attribute usage to principals:

fields identity.arn as principal, input.inputTokenCount as inTokens
| stats sum(inTokens) as totalInput, count() as calls by principal
| sort totalInput desc

CloudTrail for Bedrock:

CloudTrail tells you who invoked a model, from which IP and when. It does not store the prompt or the response.

CloudWatch Generative AI Observability: CloudWatch provides pre-built views for AI workloads. The Model Invocations view charts invocation count, latency, token counts by model, throttles and errors, and includes a table where you open a request ID to read that call's input and output. That table needs model invocation logging delivered to CloudWatch Logs. A second view covers Amazon Bedrock AgentCore agents. Alarms, dashboards and Logs Insights work on these signals as on any other.

Amazon Managed Service for Prometheus (AMP): Many container teams already standardize on Prometheus. AMP is a serverless, Prometheus-compatible metrics service. You keep the Prometheus data model and PromQL, and AWS runs the ingestion, storage, querying and availability.

CloudWatch or AMP?

⚠️ Exam Trap: Two traps show up together. First, CloudTrail is not where prompt text lives. If a scenario needs the content of model calls, the answer is model invocation logging, which is off until someone enables it. Second, AMP's alert manager is not open-source Alertmanager with every receiver. It sends only to an Amazon SNS topic in the same account, so email, Lambda and ticketing hang off that topic.

Reflection Question: A chatbot's users report intermittent failures at peak hours. InvocationThrottles spikes, InvocationServerErrors stays at zero, and a teammate wants to raise the SDK's retry count. What do you check before agreeing, and which signal tells you whether the account actually needs a larger quota?