4.3.4. Reliability Risks: Hallucinations, Data Degradation, and Drift
💡 First Principle: Generative models produce the most plausible-sounding output, not verified truth, so they can state fabrications with complete confidence. Predictive models produce the most likely answer given patterns that may no longer hold. Reliability risk is the gap between what the system says and what is true, and it must be managed by grounding outputs in verified sources, measuring accuracy continuously, and placing human judgment where errors matter.
The exam guide names three reliability risks:
| Risk | What happens | Example | Mitigations |
|---|---|---|---|
| Hallucination | A generative model produces fluent but false or unsupported content | An assistant cites a refund policy that does not exist; a summary includes a clause not in the contract | RAG grounding in approved sources, citations, contextual grounding checks (Amazon Bedrock Guardrails), Automated Reasoning checks for policy rules, constrained scope, evaluation sets, human review for high-stakes outputs |
| Data quality degradation | Input data or knowledge sources become stale, incomplete, or corrupted | A RAG knowledge base contains outdated product specs; an upstream system starts sending blank fields | Data ownership and freshness SLAs, source validation, monitoring of data pipelines and knowledge base updates |
| Model drift | Model performance decays as the world changes (2.2.3) | A demand forecast misses after a market shift | Performance monitoring, thresholds, retraining or updating plans, re-validation |
Hallucination deserves particular attention because it is inherent to how generative models work; newer models reduce but do not eliminate it. Leaders should therefore decide where hallucination is tolerable:
- Tolerable with light review: brainstorming, first drafts, internal summaries clearly labeled as AI-generated.
- Requires grounding and citations: policy answers, product information, internal knowledge search.
- Requires human verification: legal, medical, financial, or safety-related content; anything published externally under the company's name; any statement that creates a commitment to a customer.
Evaluation sets (curated questions with known correct answers) let teams measure hallucination and accuracy before launch and after every change. Combined with production sampling and user feedback, they turn reliability from an anecdote into a metric.
⚠️ Exam Trap: Hallucination is not fixed by "waiting for a better model" or by telling users the AI "may make mistakes" alone. The best answers ground responses in approved data (RAG), add verification such as contextual grounding checks and citations, restrict scope, measure with evaluation sets, and require human review where errors carry significant consequences.
Reflection Question: A customer-facing assistant told a customer they were eligible for a fee waiver that does not exist, and the customer now expects it to be honored. Which reliability risk occurred, which controls would have reduced it, and what business decision must now be made?
