3.3. Analyzing Model Performance

💡 First Principle: A model's value is measured by how well it performs on unseen data, not training data. Evaluation metrics quantify this performance—but different business problems require different metrics. Optimizing the wrong metric produces a model that's mathematically good but practically useless. The exam heavily tests your ability to select and interpret the right metric for a given scenario.

What breaks when you optimize the wrong metric? Consider a cancer screening model. Optimizing for overall accuracy on a dataset where 99% of patients are healthy produces a model that predicts "healthy" for everyone—99% accuracy, but it misses every cancer case. The correct metric here is recall (sensitivity): what fraction of actual cancer cases does the model catch? The exam presents similar scenarios and expects you to pick the right metric.

Imagine a fire alarm system. High recall means it catches every fire but also false alarms during cooking (annoying but safe). High precision means every alarm is a real fire, but it might miss some (dangerous). The business decides the trade-off—a hospital wants high recall (never miss a fire), a kitchen might accept lower recall for fewer false alarms.

⚠️ Common Misconception: High accuracy means the model is good. On imbalanced datasets, a model that always predicts the majority class achieves high accuracy while being completely useless. The exam loves this trap — if the scenario describes imbalanced classes (fraud detection, rare disease screening), accuracy is almost always the wrong metric. Look for F1, recall, or AUC-ROC instead.