Black Friday sale: an extra 40% off every course until Monday, November 30.New Year sale: an extra 30% off every course until Friday, January 1.Memorial Day sale: an extra 25% off every course until Monday, May 31.SysAdmin Day sale: an extra 25% off every course until Sunday, August 1.Back to School sale: an extra 25% off every course until Sunday, August 22.Labor Day sale: an extra 25% off every course until Monday, September 6.Black Friday sale: an extra 40% off every course until Monday, November 29.New Year sale: an extra 30% off every course until Saturday, January 1.Choose your certification →

Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.

3.6.4. ML Inference Architectures with Amazon SageMaker AI Endpoints

💡 First Principle: Match the inference option to the traffic shape and payload size, because idle capacity, not inference itself, is what usually wastes an ML budget.

Scenario: A data science team has three models to put in production: a fraud score called 300 times per second with a 50 ms target, a document classifier that receives 200 MB scanned files and may take 20 minutes each, and a churn model that scores the whole customer table once a night.

Use "Amazon Bedrock" when a managed foundation model fits. Use "Amazon SageMaker AI" endpoints when you must host your own model artifact or container, pick the instance or GPU type, or serve a classic ML model (XGBoost, a custom PyTorch model).

OptionLimitsBest forNot for
Real-timePayloads up to 25 MB; 60 s per request (8 min streaming)Steady, low-latency traffic; GPUs; VPC accessLong jobs; spiky traffic that idles most of the day
ServerlessPayloads up to 4 MB; 60 s; 1-6 GB memory; scales to zeroIntermittent or unpredictable traffic that tolerates cold starts (provisioned concurrency keeps it warm)GPU models, VPC configuration, data capture, multi-model endpoints
AsynchronousPayloads up to 1 GB; up to 1 hour per request; internal queue; scales to zeroLarge inputs, long processing, near-real-time results notified through SNSSynchronous low-latency calls
Batch transformGB-scale datasets, runs for hours or days, no persistent endpointScoring a whole dataset on a scheduleOnline requests

Packing models. Many small models can share instances on a multi-model endpoint or as inference components on one endpoint; inference components can also scale the endpoint to zero instances when idle (with a scaling policy to bring capacity back).

Integration patterns. Synchronous: client → "API Gateway" → "Lambda" (or the application directly) → InvokeEndpoint. Asynchronous: upload the input to S3, call InvokeEndpointAsync, and react to the SNS success or error notification. Offline: an EventBridge schedule starts a batch transform job that writes results to S3.

For the scenario: the fraud score uses a real-time endpoint with target-tracking autoscaling, the 200 MB classifier uses asynchronous inference (it exceeds both the real-time payload and timeout limits), and the nightly churn job uses batch transform.

⚠️ Exam Trap: Serverless inference is attractive for cost, but it does not support GPUs or VPC configuration. A scenario that needs either, or a payload above 4 MB, rules it out even if traffic is intermittent; asynchronous inference with scale to zero is usually the cost-efficient alternative.

See how it connects
Alvin Varughese
Written byAlvin Varughese
Founder•26 professional certifications earned