7.3. Reflection Checkpoint

Key Takeaways

• RAGAS provides four independent metrics that pinpoint RAG pipeline failures: Faithfulness (FM hallucinating), Context Precision (retrieval returning irrelevant chunks), Context Recall (retrieval missing needed information), Answer Relevancy (response not addressing the question).
• LLM-as-Judge scales automated evaluation to qualitative dimensions that ROUGE/BLEU cannot capture. Use a cheaper evaluator model than the application model to control evaluation costs.
• The diagnostic sequence always follows data flow: input → retrieval → context assembly → FM inference → output processing. Never assume the FM is the culprit without isolating retrieval and context quality first.
• Bedrock Model Invocation Logs are the primary forensic tool for production quality incidents — but only if enabled before the incident occurred.
• Agent looping is usually caused by ambiguous tool schemas, unrecognized error formats, or missing stopping conditions — not FM reasoning failures.
• P50 vs. P99 latency discrepancy (most requests fast, some very slow) indicates throttling or cold start rather than a systematic latency problem.

Connecting Forward

The three terminal phases complete the study guide: Phase 8 (Exam Readiness) provides strategy and a Quick Reference for exam-day review, Phase 9 (Glossary) defines all key terms, and Phase 10 (Conclusion) summarizes the journey and provides confidence checkpoints.

Self-Check Questions

• RAGAS evaluation of your financial advisory RAG bot returns: Faithfulness = 0.94, Context Precision = 0.38, Context Recall = 0.85, Answer Relevancy = 0.91. Write a specific, prioritized remediation plan naming the exact AWS services and configuration changes to address the root cause.
• Your Bedrock Agent handles insurance claim queries. It was working correctly last week but now loops 8–12 times before answering "I was unable to complete this task." No code changes were deployed. What are the five most likely causes, and how would you diagnose each one in under 15 minutes?