5.2.3. Troubleshooting Latency and Scaling Issues
💡 First Principle: Latency in ML systems compounds across the prediction pipeline—preprocessing, model inference, and postprocessing each contribute. A 200ms latency target means each stage gets a fraction of that budget, and the bottleneck is often not where you expect. Diagnose where the time goes before changing capacity.
SageMaker AI splits the time an endpoint request takes into parts you can measure separately:
- Network latency — between the client and the SageMaker Runtime API. It appears in neither endpoint metric; measure it at the client.
OverheadLatency— time SageMaker AI adds around the model: moving the request to the container and the response back. It varies with payload size, request frequency, and authentication and authorization of the request.ModelLatency— time the model container takes to respond, including the inference itself and any preprocessing or postprocessing code that runs inside the container.
Common latency patterns and how to read them:
Slow inside the container. If ModelLatency dominates, the work is in your model or your container code. Check whether the instance is saturated (CPUUtilization, GPUUtilization, MemoryUtilization). If it is, scale out or move to a larger or accelerated instance family; SageMaker Inference Recommender can load-test candidate instance types. If it is not saturated, profile the container code: garbage collection pauses, per-request model reloading, or inefficient preprocessing produce slow requests on idle hardware.
Slow around the container. If OverheadLatency dominates, look at payload size and request pattern before touching the model. Very large payloads (for example, uncompressed images) inflate overhead; batching several records per request or reducing the payload helps.
Slow only under load. When latency rises with traffic, capacity is lagging behind demand. The commonly used predefined scaling metric, SageMakerVariantInvocationsPerInstance, reacts to request counts, and new instances need time to launch and load the model. Use scheduled scaling for predictable peaks, step scaling for sharp spikes, and a scaling metric that reflects the real bottleneck—for GPU-bound models, target tracking on GPUUtilization with a customized metric specification.
Slow only on the first request. New instances and serverless endpoints must load the model before serving. A minimum instance count keeps at least one warm instance for real-time endpoints; choosing between serverless, real-time and asynchronous endpoints based on traffic pattern is a deployment decision covered in Phase 4.
| Symptom | Likely Cause | First Check | Fix |
|---|---|---|---|
High ModelLatency, instance saturated | Under-provisioned compute | CPU/GPU utilization | Scale out, larger or accelerated instance |
High ModelLatency, instance idle | Container code path | Profiling, container logs | Fix code path, avoid per-request loading |
High OverheadLatency | Large payloads or request pattern | Payload size | Compress, batch records |
| Latency rises with traffic | Scaling lag | Scaling activity vs InvocationsPerInstance | Scheduled or step scaling, better metric |
| p99 far above p50, low utilization | Intermittent software stalls | Traces and logs for slow requests | Profile the slow path |
⚠️ Exam Trap: When a question describes latency problems, don't immediately jump to "use a bigger instance." First determine where the latency originates—ModelLatency versus OverheadLatency, saturated versus idle hardware. A bigger instance cannot fix a payload problem or a stalled code path. The exam rewards answers that diagnose before prescribing.
Reflection Question: A real-time endpoint serving a computer vision model shows P99 latency of 3 seconds while the P50 is 150ms. What does this discrepancy suggest, and how would you investigate and fix it?
