2.4.2. Machine Learning Value and Security Considerations
💡 First Principle: Machine learning models learn from data, which creates both their value and their risk. The value: models can recognize patterns and make predictions humans couldn't. The risk: if training data or model access is compromised, the model's value is compromised too.
Understanding the ML lifecycle helps you identify security touchpoints and make informed decisions about when ML adds value:
The ML lifecycle in detail:
- Problem definition — What business outcome are we predicting or classifying? Without a clear problem, ML produces impressive technology that solves nothing.
- Data collection — Gather relevant, representative training data. Where does it come from? Is it sensitive? Is it sufficient? Put access controls, anonymization, and data governance in place from this point: gaps here (uncontrolled access, sensitive data not identified or anonymized, privacy violations) are easy to miss while teams focus on model performance, and every later stage inherits them.
- Data preparation — Clean, label, and structure the data. This is often 60-80% of the total effort—messy data produces unreliable models.
- Model training — The algorithm learns patterns from the prepared data. Different algorithms suit different problems.
- Evaluation — Test the model against held-out data. Measure accuracy, fairness, and reliability. If insufficient, iterate.
- Deployment — Put the model into production where it serves real users and decisions. Each time the deployed model processes a request and produces an output, that is inference—the run-time stage, as distinct from training.
- Monitoring and retraining — Models degrade over time as real-world data changes ("model drift"). Continuous monitoring detects degradation; retraining corrects it. Monitoring is a security control too: drift and new attack techniques (such as prompt injection or poisoned data) emerge after launch, so a model that was safe at deployment can become vulnerable.
Key insight for business leaders: The ML lifecycle is iterative, not linear. Models require ongoing investment in monitoring and retraining—it's not a "build once, done forever" proposition. Budget for ongoing maintenance, not just initial development. This matters even if you only buy pre-built AI: knowing the lifecycle lets you make informed buy-vs-build decisions and ask vendors the right questions—what data trained the model, how it is evaluated, and how it is monitored and updated.
Security considerations at each ML lifecycle stage:
| Stage | Security Concern | Mitigation |
|---|---|---|
| Training data | Contains sensitive information | Data anonymization, access controls |
| Model weights | Represent significant IP investment | Access restrictions, encryption |
| Prompts | May contain confidential information | Secure transmission, logging policies |
| Outputs | May reveal protected information | Access controls, content filtering |
Prompt logs are sensitive data too. Keep them long enough for security monitoring and compliance, but no longer than needed—set retention by balancing those needs against privacy obligations and data minimization, rather than keeping everything forever or deleting immediately.
⚠️ Exam Trap: The exam may present scenarios where users want to share AI models externally. Consider what the model "knows" from training—sharing a model trained on confidential data could expose that information. "It's only weights" is not a safe assumption: models can memorize patterns (sometimes specific records) from their training data, and targeted prompting or inversion attacks can extract them. That is why training data (the raw records) and model weights (the patterns learned from them) each need their own protection.
Reflection Question: Your organization trained a custom model on internal sales data. A partner company asks if they can use this model. What concerns should you raise?
