3.6.4. ML Inference Architectures with Amazon SageMaker AI Endpoints
💡 First Principle: Match the inference option to the traffic shape and payload size, because idle capacity, not inference itself, is what usually wastes an ML budget.
Scenario: A data science team has three models to put in production: a fraud score called 300 times per second with a 50 ms target, a document classifier that receives 200 MB scanned files and may take 20 minutes each, and a churn model that scores the whole customer table once a night.
Use "Amazon Bedrock" when a managed foundation model fits. Use "Amazon SageMaker AI" endpoints when you must host your own model artifact or container, pick the instance or GPU type, or serve a classic ML model (XGBoost, a custom PyTorch model).
| Option | Limits | Best for | Not for |
|---|---|---|---|
| Real-time | Payloads up to 25 MB; 60 s per request (8 min streaming) | Steady, low-latency traffic; GPUs; VPC access | Long jobs; spiky traffic that idles most of the day |
| Serverless | Payloads up to 4 MB; 60 s; 1-6 GB memory; scales to zero | Intermittent or unpredictable traffic that tolerates cold starts (provisioned concurrency keeps it warm) | GPU models, VPC configuration, data capture, multi-model endpoints |
| Asynchronous | Payloads up to 1 GB; up to 1 hour per request; internal queue; scales to zero | Large inputs, long processing, near-real-time results notified through SNS | Synchronous low-latency calls |
| Batch transform | GB-scale datasets, runs for hours or days, no persistent endpoint | Scoring a whole dataset on a schedule | Online requests |
Packing models. Many small models can share instances on a multi-model endpoint or as inference components on one endpoint; inference components can also scale the endpoint to zero instances when idle (with a scaling policy to bring capacity back).
Integration patterns. Synchronous: client → "API Gateway" → "Lambda" (or the application directly) → InvokeEndpoint. Asynchronous: upload the input to S3, call InvokeEndpointAsync, and react to the SNS success or error notification. Offline: an EventBridge schedule starts a batch transform job that writes results to S3.
For the scenario: the fraud score uses a real-time endpoint with target-tracking autoscaling, the 200 MB classifier uses asynchronous inference (it exceeds both the real-time payload and timeout limits), and the nightly churn job uses batch transform.
⚠️ Exam Trap: Serverless inference is attractive for cost, but it does not support GPUs or VPC configuration. A scenario that needs either, or a payload above 4 MB, rules it out even if traffic is intermittent; asynchronous inference with scale to zero is usually the cost-efficient alternative.