3.3.2. Model Bias Detection and Explainability with SHAP
💡 First Principle: A model can amplify biases present in training data, producing predictions that systematically disadvantage certain groups. Post-training bias metrics measure this on the model's predictions, and SHAP explains which features drove each prediction. The exam tests which metric answers which question and how to interpret the output; the exam guide phrases these skills as "explain model outputs" and "bias detection" without naming a tool.
📌 Availability note: Amazon SageMaker Clarify packaged these metrics and SHAP explanations as processing jobs, but it is no longer open to new customers. Existing customers can continue to use it as normal; AWS plans no new features. AWS's recommended path for a new team is to compute the post-training metrics from predictions and ground truth with pandas and scikit-learn, compute feature attributions with the open-source SHAP library (the same engine Clarify uses), run both as a SageMaker Pipelines processing step, and log the results to a SageMaker AI MLflow App. For foundation models, AWS points to the fmeval library or Amazon Bedrock Evaluations, which do not replace tabular bias and explainability analysis.
Post-Training Bias Metrics (different from the pre-training metrics in 2.3.1):
| Metric | What It Measures | Concern If |
|---|---|---|
| Disparate Impact (DI) | Ratio of positive outcomes between groups | Far from 1.0 |
| Conditional Demographic Disparity (CDD) | Disparity conditioned on other attributes | Significant after controlling for legitimate factors |
| Counterfactual Fliptest | Whether changing a protected attribute changes the prediction | High flip rate |
SHAP Values (Shapley Additive Explanations): SHAP explains individual predictions—which features contributed most and in which direction. This is critical for interpretability: you can tell a loan applicant not just that they were rejected, but which factors (income, credit history, employment length) drove the decision and by how much.
Partial Dependence Plots (PDPs): Show how a single feature's value affects predictions on average, across the entire dataset. Useful for understanding the learned relationship between a feature and the target.
⚠️ Exam Trap: Pre-training bias detection (data-level) and post-training bias detection (model-level) use different metrics. Pre-training uses CI and DPL (data distribution metrics). Post-training uses DI and CDD (prediction outcome metrics). A question about "bias in the training data" points to pre-training metrics. A question about "biased predictions" points to post-training metrics.
⚠️ Exam Trap: Amazon Bedrock Evaluations and Amazon Bedrock Guardrails evaluate and filter foundation-model output. They do not compute Disparate Impact or SHAP attributions for a tabular classifier. A scenario about explaining a credit model's decisions points to SHAP, not to a Bedrock feature. And for a team new to SageMaker Clarify, "run a Clarify processing job" is no longer an available choice.
Reflection Question: A lending model has DPL of 0.0 (training data is perfectly balanced between demographic groups) but Disparate Impact of 0.6 (Group B’s approval rate is only 0.6 of Group A’s, so Group A is approved at about 1.67× Group B’s rate). How is this possible, and what does it mean?
