3.4.7. Monitoring RAG Systems and Retrieval Accuracy
💡 First Principle: A RAG system can fail in two separable places — retrieval (the right context was not found) and generation (the model misused good context). Monitoring must measure both separately, or every problem looks like "the model hallucinated" and gets the wrong fix.
| Failure point | Metric (Bedrock RAG evaluation) | What a low score means | Typical fix |
|---|---|---|---|
| Retrieval | Context relevance | Retrieved chunks are off-topic for the question | Better chunking, metadata filters, hybrid search, reranking |
| Retrieval | Context coverage (requires ground truth) | Retrieved chunks miss information the reference answer needs | Raise numberOfResults, hierarchical chunking, better embedding model, query decomposition |
| Generation | Faithfulness | Answer includes information not supported by retrieved chunks | Stricter prompt template, contextual grounding check, different model |
| Generation | Citation precision / citation coverage | Citations are wrong, or claims lack citations (use the two together) | Prompt template with citation instructions, smaller cleaner chunks |
| Generation | Correctness, completeness, helpfulness, logical coherence | Answer quality problems given the context | Prompt engineering, model choice |
| Responsible AI | Harmfulness, stereotyping, refusal | Unsafe, biased or evasive responses | Guardrails, prompt changes |
Bedrock RAG evaluation jobs come in two types: retrieve only (scores the retrieved context) and retrieve and generate (scores the final answers too). They can evaluate an Amazon Bedrock knowledge base or an external RAG system whose responses you bring as data, and each metric is an average score between 0 and 1 over the prompt dataset.
Continuous monitoring in production:
- Scheduled RAG evaluation jobs over a fixed "golden" question set after every knowledge base sync, chunking change or embedding change — a regression in context coverage after re-ingestion is caught before users notice.
- Knowledge base ingestion logs (delivered to CloudWatch Logs, Amazon S3 or Amazon Data Firehose) show per-document status such as
RESOURCE_IGNORED,EMBEDDING_FAILEDorINDEXING_FAILED— documents that never made it into the index cannot be retrieved. - Model invocation logging (disabled by default) captures prompts, responses and token counts to CloudWatch Logs or S3 for sampling, offline evaluation and cost analysis.
- Amazon Bedrock Guardrails contextual grounding check scores each response for grounding (supported by the source) and relevance (answers the query) against thresholds you set, and blocks responses below them — an online, per-request control rather than an offline metric.
- CloudWatch generative AI observability dashboards show model invocation counts, latency, token usage and errors, alongside agent traces.
- User feedback signals (thumbs down, reformulated queries) sampled into human review queues.
⚠️ Exam Trap: Context coverage requires ground-truth reference texts; context relevance does not. If a scenario says "we have no reference answers yet," coverage cannot be the metric they start with.
⚠️ Exam Trap: When answers are wrong and the retrieved chunks are irrelevant, changing the generator model will not help — fix retrieval first. When chunks are relevant but the answer contradicts them, the problem is generation (faithfulness), and grounding checks or prompt changes are the fix.
