5.2.3. Troubleshooting Latency and Scaling Issues
💡 First Principle: Latency in ML systems compounds across the prediction pipeline—preprocessing, model inference, and postprocessing each contribute. A 200ms latency target means each stage gets a fraction of that budget, and the bottleneck is often not where you expect. Scaling issues arise when one stage can't keep up with the others.
Common latency problems in SageMaker endpoints and their fixes:
Cold starts occur on serverless endpoints and newly auto-scaled instances when the model needs to load into memory. For serverless endpoints, keep-alive traffic or provisioned concurrency can mitigate this. For auto-scaled endpoints, configuring a minimum instance count ensures at least one warm instance is always ready.
Model size bottlenecks arise when large models (multi-GB) take significant time to load and run inference. Solutions include model compilation with SageMaker Neo (which optimizes the model for the target hardware), model compression (pruning, quantization), or switching to a GPU instance that can process the model faster.
Scaling lag occurs when traffic spikes faster than auto scaling can respond. The default SageMaker scaling policy reacts to metrics like InvocationsPerInstance—but it takes time to launch new instances and load models. Configuring more aggressive step scaling policies, setting predictive scaling based on historical patterns, or pre-warming instances before anticipated spikes can address this.
Service quota limits are a silent killer. Each AWS account has default limits on the number of concurrent SageMaker endpoints, instances, and other resources. When you hit a quota, new instances can't launch and scaling fails. The fix is requesting quota increases before they're needed.
| Symptom | Likely Cause | Fix | AWS Tool |
|---|---|---|---|
| High latency on first request | Cold start | Provisioned concurrency, min instances | Serverless config / auto scaling |
| Latency increases under load | Under-provisioned | Scale out, larger instance | Auto scaling, Inference Recommender |
| Latency spikes at consistent times | Predictable traffic pattern | Scheduled scaling | Auto scaling policies |
| Scaling fails silently | Service quota exhaustion | Request quota increase | Service Quotas, Trusted Advisor |
| Consistent high latency | Model too large for instance | Compile with Neo, upgrade instance | SageMaker Neo, Inference Recommender |
⚠️ Exam Trap: When a question describes latency problems, don't immediately jump to "use a bigger instance." First determine where the latency originates—is it cold start, model inference time, or data preprocessing? The exam rewards answers that diagnose before prescribing. X-Ray traces or CloudWatch ModelLatency vs. OverheadLatency metrics help isolate the cause.
Reflection Question: A real-time endpoint serving a computer vision model shows P99 latency of 3 seconds while the P50 is 150ms. What does this discrepancy suggest, and how would you investigate and fix it?
