2.1.3. Privacy and Security
💡 First Principle: Privacy and security mean protecting the personal data an AI system uses and defending the system itself against misuse. AI is data-hungry, and that data is often sensitive, so this principle covers both what you collect and keep (privacy) and who can access or attack the system (security).
In practice this looks like collecting only the data you need (data minimization), securing it in transit and at rest, controlling access, and guarding the model against attacks. A modern, AI-specific concern here is the prompt injection attack — including cross-prompt injection (XPIA) — where malicious instructions hidden in content try to hijack a model or agent's behavior. Microsoft Foundry includes built-in guardrails aimed at reducing unsafe outputs and mitigating these injection risks, which is why this principle resurfaces in the implementation phases.
⚠️ Exam Trap: Privacy and security are related but distinct. Privacy is about the appropriate handling of personal data (consent, minimization, retention); security is about protecting systems and data from threats (access control, attack mitigation). A scenario about leaked customer records leans privacy; one about an attacker manipulating a model leans security.
Reflection Question: A team wants to train a support chatbot on real customer-service transcripts. Name one privacy concern and one security concern they should address before doing so.
