3.2.1. Why AI Output Must Be Validated
💡 First Principle: You validate because the model optimizes for plausibility, and plausibility and correctness diverge in exactly the cases that matter — subtle bugs, security flaws, and code that compiles but does the wrong thing.
Validation exists because of everything in 3.1.1: hallucination, insecure patterns, bias, and confident-but-wrong output. The dangerous failures are the fluent ones — code that looks right, compiles, and passes a glance, yet is subtly incorrect or insecure.
What thorough validation covers:
- Correctness — does it actually meet the requirement, including edge cases?
- Security — does it introduce vulnerabilities or unsafe defaults?
- Performance — is it efficient enough for its context?
- Licensing / IP — could it match public code in a way that matters?
- Fit — does it follow team conventions and integrate cleanly?
A scenario: Copilot generates a function that compiles and passes the happy path but mishandles empty input, producing a runtime error in production. "It compiled" gave false confidence. Validation means testing the edges, not just the build.
❌ Common Mistake: Equating a clean build or a passing happy-path test with "validated." The subtle, expensive failures live in the cases you didn't check.
⚠️ Exam Trap: If a scenario describes a developer who accepted a suggestion that "looked fine" and shipped a defect, the corrective action is to strengthen validation (tests, review, scanning) — not to choose a different model or trust Copilot more.
Reflection Question: Why are the most dangerous AI errors often the ones that compile and pass a quick look, and what does that imply about how thoroughly you should validate?
