5.1.2. Production Model Monitoring: Baselines, Data Capture and Drift Checks
💡 First Principle: Production monitoring works by continuously comparing production data against a baseline you define at deployment time. The quality of your monitoring depends entirely on the quality of your baseline—a poorly constructed baseline produces either false alarms (too sensitive) or missed detections (too lenient). That holds whichever tool runs the comparison.
📌 Availability note: Amazon SageMaker Model Monitor, the managed service that long ran these checks, is no longer open to new customers. Existing customers can continue to use it as normal; AWS continues security and availability improvements but plans no new features. The MLA-C02 exam guide phrases the skill as "create model performance baselines and implement drift detection" and names no service. For a new team, AWS recommends its open-source Amazon SageMaker AI monitoring solutions (published in the aws-samples GitHub organization and run entirely inside your own account) together with Amazon CloudWatch and Amazon QuickSight.
The recommended pattern has four steps, the same shape as the classic Model Monitor workflow:
Step 1: Enable Data Capture on your SageMaker endpoint. Data Capture is still an endpoint feature, and the reference solutions consume it: sampled inference requests and responses are stored in S3. You configure what percentage of traffic to capture—100% for low-volume endpoints, a sample for high-volume ones.
Step 2: Create a Baseline from the training data, which becomes the reference dataset: statistics and distribution profiles for each feature, plus the model quality metrics from evaluation. The baseline becomes the "normal" against which production data is measured. The reference solutions keep it in S3 and log every drift result to the same SageMaker AI MLflow App as the training run, so drift results share the model's lineage.
Step 3: Run drift checks on a schedule. An Amazon EventBridge schedule or an S3 event invokes AWS Lambda, or a SageMaker Pipeline runs on an EventBridge schedule. Evidently AI compares captured production data with the reference dataset: Population Stability Index and Kolmogorov-Smirnov statistics per feature for data drift, plus missing-value and outlier checks for data quality. When ground truth arrives it computes accuracy, precision, recall, F1 and ROC-AUC. Results are logged to MLflow, and the QuickSight governance solution also keeps them in an Amazon Athena (Apache Iceberg) data lake for trend analysis.
Step 4: Alert and act. Amazon SNS notifies the team when a threshold is crossed, and the same event can start a retraining pipeline. Amazon CloudWatch adds endpoint metrics (latency, errors, CPU and GPU utilization) and custom metrics with anomaly detection alarms. Amazon QuickSight dashboards give governance teams the drift and performance trends.
Production monitoring covers four dimensions. They keep the names Model Monitor gave them, and each has a recommended equivalent:
| Dimension | What It Checks | Baseline Source | Recommended Approach (new teams) | Exam Signal |
|---|---|---|---|---|
| Data Quality | Feature statistics, missing values, data types, distribution shift | Training dataset statistics | Evidently data drift and data summary reports on captured data | "Feature distributions shifted" |
| Model Quality | Accuracy, precision, recall, F1 vs. baseline | Model evaluation metrics | Evidently classification report after ground truth is joined to predictions, or CloudWatch custom metrics | "Model performance degraded" (requires ground truth) |
| Bias Drift | Fairness metrics across groups | Post-training bias metrics on the evaluation set (pre-training metrics describe the dataset only) | Per-segment metrics (selection rate, true and false positive rate gaps) on a schedule, logged to MLflow, with an SNS alert on a threshold | "Disproportionate impact on group" |
| Feature Attribution Drift | SHAP importance distributions | SHAP feature importance on the evaluation set | SHAP computed on recent traffic and compared with the baseline ranking | "Feature importance rankings changed" |
⚠️ Exam Trap: Model Quality monitoring requires ground truth labels for production data to compute accuracy metrics. If the scenario doesn't mention ground truth availability, Model Quality monitoring can't be used—only Data Quality monitoring (which compares feature distributions without needing labels). Labels often arrive late, so the reference solutions join delayed ground truth to predictions by inference ID. The exam tests this prerequisite.
⚠️ Exam Trap: For a team that is new to SageMaker Model Monitor, for example one building in a new AWS account, "create a Model Monitor monitoring schedule" is no longer an available answer because the service is closed to new customers. Look for the pattern instead: Data Capture, a baseline, a scheduled drift check, and an alert. Existing customers who already run Model Monitor can keep it.
Reflection Question: A team has deployed drift monitoring on an endpoint but is getting too many false positive alerts. What are two approaches to reduce noise without missing real drift?
