3.1.1. Risks and Limitations of Generative AI
💡 First Principle: Every risk of generative AI traces back to the same root: the model produces plausible output, not verified output. Hallucinations, insecure code, bias, and staleness are all faces of "plausible ≠ correct."
The exam expects you to describe the concrete risks and limitations of generative tools like Copilot:
- Hallucination — the model can invent plausible-looking but nonexistent APIs, functions, or facts. It is filling in what looks right, not retrieving truth.
- Insecure or low-quality suggestions — generated code can contain vulnerabilities (injection, weak crypto, unsafe defaults) because the training data contained them too.
- Bias — patterns in training data can surface as biased or non-inclusive output.
- Knowledge cutoff and context limits — the model doesn't know about very recent libraries or your whole codebase unless that context is supplied; it reasons within a finite context window.
- Over-reliance / automation bias — humans tend to trust fluent, confident output, which is precisely when unchecked errors slip through.
A scenario the exam likes: Copilot suggests code using a library method that doesn't exist. The lesson isn't "Copilot is broken" — it's that hallucination is an inherent limitation, and the developer must verify against real documentation.
💡 Key Point: "Confident" is not "correct." The fluency of LLM output is a risk factor, because it makes errors persuasive.
⚠️ Exam Trap: Beware answers implying Copilot's suggestions are inherently secure or correct "because it learned from real code." Training imparts patterns, including bad ones. The presence of a risk is not a malfunction; it's a property of the technology.
Reflection Question: Why are hallucination, insecure suggestions, and bias all consequences of the same underlying fact about how generative models work?
