4.4.3. Hosting Foundation Models on SageMaker AI and Scaling GPUs
💡 First Principle: A large model is expensive per instance and slow to start, so the hosting design has two goals: pack models onto accelerators efficiently, and scale on a signal that reacts before users feel the queue.
Inference components decouple a model from the endpoint that hosts it. Each component declares the resources one copy needs (NumberOfAcceleratorDevicesRequired, NumberOfCpuCoresRequired, MinMemoryRequiredInMb) and a CopyCount. You can place several FMs on one endpoint, update each independently, and scale each component's copies on its own. With managed instance scaling on the production variant, SageMaker AI adds or removes instances to fit the copies you ask for.
Large model inference (LMI) containers bundle model servers and optimized backends (for example vLLM and TensorRT-LLM) for LLM serving. Tensor parallelism shards one model copy across several GPUs of an instance. option.tensor_parallel_degree takes a value from 1 up to the number of GPUs, or max to shard across all of them; when the degree is lower than the GPU count, LMI starts as many model copies as fit (4 GPUs at degree 2 run two copies of 2 GPUs each).
Startup settings that break big deployments. A multi-GB model can exceed default timeouts before the container ever answers a health check. The production variant exposes ModelDataDownloadTimeoutInSeconds and ContainerStartupHealthCheckTimeoutInSeconds (each 60 to 3,600 seconds) and VolumeSizeInGB (up to 512 GB of EBS storage). A deployment that fails with a health-check timeout while CloudWatch logs show the model still loading is a timeout problem, not an instance-size problem.
Scaling metrics for generative AI. LLM requests are long and concurrency-bound, so invocation counts per minute lag reality. SageMaker AI offers high-resolution concurrency metrics emitted every 10 seconds:
| Metric (predefined type) | Scope | Use when |
|---|---|---|
SageMakerVariantInvocationsPerInstance | Variant | Short, uniform requests (classic ML) |
SageMakerInferenceComponentInvocationsPerCopy | Inference component | Per-copy scaling of short requests |
SageMakerVariantConcurrentRequestsPerModelHighResolution | Variant | Streaming or long-running LLM requests on a single-model endpoint |
SageMakerInferenceComponentConcurrentRequestsPerCopyHighResolution | Inference component | Streaming or long-running LLM requests on multi-model inference component endpoints |
AWS reports scale-up detection up to 6x faster with the concurrency metrics than with invocation-per-instance scaling.
Scale to zero. An endpoint can scale in to zero instances only if it hosts inference components and the variant's ManagedInstanceScaling has MinInstanceCount of 0. Register each component with a minimum capacity of 0. Scaling out from zero requires a step scaling policy triggered by a CloudWatch alarm on NoCapacityInvocationFailures; provisioning takes several minutes and invocations fail meanwhile, so scale to zero suits dev/test and tolerant batch-like callers, not interactive traffic.
⚠️ Exam Trap: Target tracking alone cannot bring an endpoint back from zero instances, because there is no instance emitting the tracked metric. The distinguishing detail in a scale-to-zero scenario is the step scaling policy on NoCapacityInvocationFailures.
