5.4. Operating, Monitoring and Securing Generative AI Workloads
💡 First Principle: A foundation model (FM) or agent in production fails in ways a classical model cannot. A regression model returns a number; an FM returns prose that can be fluent and wrong, cut off halfway, unsafe, or ten times longer (and costlier) than it needed to be. An agent adds a second failure surface: the tools it calls. Operating generative AI therefore needs three extra instruments on top of Sections 5.1 to 5.3: quality monitoring that judges the text itself, token-level cost accounting, and content safeguards that inspect what goes in and comes out.
Sections 5.1 to 5.3 still apply. Baseline drift monitoring still watches tabular endpoints, CloudWatch still collects metrics, and IAM, VPCs and KMS still guard the data. What this section adds is the layer those tools cannot see. CloudWatch can report that an endpoint answered in 900 ms; it cannot tell you the answer was a hallucination. Cost Explorer can report the monthly Amazon Bedrock bill; it cannot tell you which prompt template or which agent loop burned the tokens unless you attribute usage deliberately.
What goes wrong without this layer? A support assistant's answers drift after a model version change and nobody notices, because latency and error rates are flat. An agent's calls to an order-lookup tool start failing and the model quietly apologises instead of erroring, so the 5xx dashboard stays green. A retrieval pipeline re-embeds an unchanged document corpus every night, and the embedding bill grows faster than the traffic. Every one of these incidents is invisible to infrastructure monitoring alone.
The seven subsections follow the same order as that list: see the workload (5.4.1), judge its quality (5.4.2), watch its agents (5.4.3), control what it costs (5.4.4 and 5.4.5), then secure how it is built and called (5.4.6) and what it is allowed to say (5.4.7).
⚠️ Common Misconception: "Generative AI monitoring is just CloudWatch metrics for Amazon Bedrock." Metrics tell you how much and how fast. They say nothing about whether an answer was correct, grounded, safe or complete. Quality needs evaluations, content needs guardrails, and cost attribution needs logs or tagged inference profiles. The exam tests whether you pick the instrument that can actually observe the failure described.
