5.1.1. Data Drift and Model Drift Detection

💡 First Principle: When the real world changes, the statistical patterns your model learned become stale. Data drift means the inputs have changed; model drift means the relationship between inputs and outputs has changed. Distinguishing between them determines whether you need to retrain the model, fix the data pipeline, or both.

Data drift and model drift are related but distinct problems, and the exam tests whether you know the difference:

Data drift (also called feature drift or covariate shift) occurs when the distribution of input features changes. For example, if your model was trained on data where average customer age was 35 and production traffic shifts to average age 55, that's data drift. The model hasn't degraded—the world moved away from the training data.

Model drift (also called concept drift) occurs when the relationship between features and the target variable changes. In fraud detection, new fraud techniques emerge that create patterns the model has never seen. The features may look the same, but what constitutes "fraudulent" has changed.

Prediction drift occurs when the distribution of model outputs changes, even if inputs appear stable. This often signals a subtler issue—perhaps a preprocessing bug is silently transforming data differently, or an upstream system changed its output format.

A drift monitor detects drift by comparing production data against a baseline—a statistical profile of the training data. When feature distributions shift beyond a configurable threshold (measured with statistics such as the Population Stability Index or the Kolmogorov-Smirnov test), it raises an alert. Bias drift applies the same idea to fairness: recompute post-training bias metrics per demographic segment on recent predictions and alert when the gap from the baseline widens. Existing AWS customers may run these checks with SageMaker Model Monitor (and Clarify for bias drift); both are closed to new customers, and 5.1.2 covers the approach AWS recommends instead.

⚠️ Exam Trap: A question describing "model accuracy degradation over time" is testing drift detection (production monitoring against a baseline), not training-time debugging. Convergence debugging (MLflow, TensorBoard and CloudWatch training metrics, or SageMaker Debugger for existing customers) is for training-time issues. Drift monitoring is for production-time drift. The exam frequently presents this distinction.

Reflection Question: Your e-commerce recommendation model's click-through rate has dropped 15% over two months. How would you determine whether this is caused by data drift, concept drift, or a data pipeline issue? Which AWS tools would you use?