5.4.2. Monitoring FM Quality with Amazon Bedrock Evaluations
💡 First Principle: An FM's quality cannot be read from a confusion matrix, because most outputs have no single correct label. Quality has to be judged — by a metric that compares against a reference, by another model acting as a judge, or by people. Monitoring FM quality in production means running those judgements repeatedly, on data that looks like current traffic, and alarming when the scores move.
Amazon Bedrock evaluations offers several job types. The exam tests which one fits the constraint in the scenario:
| Evaluation type | How it scores | Best when | Watch out for |
|---|---|---|---|
| Automatic (programmatic) | Computed metrics for a task type: accuracy, robustness, toxicity | Quick benchmark on a standard task, using a built-in or your own prompt dataset | Metrics are task-specific (for example BERTScore for summarization, F1 for question answering) |
| Model as a judge (LLM-as-a-judge) | An evaluator model scores each response and explains the score | Open-ended text where you care about correctness, completeness, faithfulness, helpfulness, harmfulness and similar qualities | Needs two models: a generator and an evaluator |
| Human-based | Your own work team rates or ranks responses | Subjective or domain-specific judgements, brand voice, final sign-off | Slow and expensive; does not scale to continuous monitoring |
| RAG evaluation | LLM-based metrics for retrieval and for retrieve-and-generate | Checking a knowledge base retrieves the right context and answers from it | Dataset must include ground truth for the expected retrieved text and responses |
Built-in judge metrics include Correctness, Completeness, Faithfulness, Helpfulness, Logical coherence, Relevance, Following instructions, Professional style and tone, plus the responsible-AI metrics Harmfulness, Stereotyping and Refusal. You can also define custom metrics with your own judging prompt.
Two capabilities turn evaluation from a one-off test into a monitoring tool:
- Bring your own inference responses. A judge-model or RAG evaluation job can score responses you supply instead of invoking a model. Bedrock skips the invoke step and evaluates the data directly. That means you can evaluate responses captured from production — including responses from models hosted outside Amazon Bedrock.
- Every generator type is supported. Foundation models, customized and imported models, Marketplace models, prompt routers and Provisioned Throughput models can all be the generator, so the same evaluation can compare a candidate model or router against the one in production.
An FM drift pipeline. The statistical drift baselines of 5.1 do not apply to free text, but the pattern carries over: capture, baseline, compare on a schedule, alarm.
A scheduled job (Step Functions, SageMaker AI Pipelines or EventBridge Scheduler invoking Lambda) samples recent records from invocation logs, builds a prompt dataset that includes the captured responses, runs an evaluation with the same metrics as the original baseline, and publishes the scores as custom CloudWatch metrics. A drop in Faithfulness or Correctness against the baseline is the generative equivalent of a data drift violation. Keep the evaluator model and metric definitions fixed between runs; changing the judge changes the ruler.
The drift monitoring skills in 5.1 still cover tabular and embedding-input models — for example, drift in the distribution of input features to a classifier that sits behind the FM. Use both where a system has both.
⚠️ Exam Trap: A statistical drift monitor (SageMaker Model Monitor for existing customers, or an Evidently drift report) is not the answer for "detect that a chatbot's answers are becoming less accurate". It compares statistical distributions of structured data and needs ground-truth labels for model quality. Judging generated text needs an evaluation — model as a judge or human — not a statistics baseline.
⚠️ Exam Trap: A human evaluation is the most trustworthy judgement, but it is the wrong answer when the scenario asks for continuous or automated monitoring with least operational overhead. Scheduled judge-model evaluations scale; human panels are for calibration and sign-off.
