4.1.4. Choosing Where a Model Runs: SageMaker AI, Amazon Bedrock or Your Own Compute
💡 First Principle: Every model needs somewhere to run, and the three AWS answers differ in one dimension: how much of the serving stack you manage. Amazon Bedrock manages everything and bills by token. SageMaker AI manages the platform but lets you choose model, container and instance. Amazon ECS, Amazon EKS, AWS Lambda and Amazon EC2 give you full control and full responsibility. Start from the most-managed option (the least for you to operate) that satisfies the constraints in the scenario.
| Requirement in the scenario | Best fit | Why |
|---|---|---|
| Use a catalog FM (for example Anthropic Claude, Amazon Nova, Meta Llama) with no infrastructure | Amazon Bedrock on-demand | Serverless, pay per token, integrates with Knowledge Bases, Guardrails and AgentCore |
| Serve open weights you fine-tuned elsewhere in a supported architecture, pay only when used | Bedrock Custom Model Import | Serverless hosting for imported Hugging Face format weights |
| Specialized model not in the serverless catalog, but you still want Bedrock APIs and tools | Amazon Bedrock Marketplace | Deploys to a SageMaker AI managed endpoint, invoked through Bedrock APIs |
| Your own classical ML model, custom inference code, or a specific GPU or Inferentia instance | SageMaker AI endpoint (real-time, serverless, asynchronous or batch transform) | Full control of container and instance with managed hosting, scaling and deployment guardrails |
| Many FMs sharing GPUs, each scaled independently | SageMaker AI inference components | Per-model accelerator allocation and copy counts on one endpoint |
| Inference must live inside an existing Kubernetes platform or microservice mesh | Amazon EKS or Amazon ECS | Reuse existing orchestration, networking and tooling |
| Lightweight model or pre/post-processing step, sporadic calls | AWS Lambda | Pay per invocation; .zip package contents up to 250 MB unzipped (including layers), container images up to 10 GB |
The four questions that decide it:
- Is the model already a managed FM? If yes, Bedrock is the default; moving it to SageMaker AI adds instance management for no benefit.
- Do you need to control the runtime? Custom code, custom containers or a specific accelerator push you to SageMaker AI.
- What is the traffic shape? Token-based pricing favours spiky, low-volume use; always-on instances favour high, steady volume where per-hour cost amortises well.
- Where does the rest of the application run? An organisation standardised on EKS may accept the extra operational load to keep one platform.
⚠️ Exam Trap: "Most control" is rarely the scored answer. If a scenario asks for the LEAST operational overhead and the model is available in Amazon Bedrock, deploying the same model on SageMaker AI GPU instances or EKS is a distractor even though it would work. Choose SageMaker AI when the stem gives a concrete reason Bedrock cannot meet, such as an unsupported architecture, custom inference code or a required instance type.
Reflection Question: A retailer wants a product-description generator using a model available in Amazon Bedrock, plus a demand-forecasting XGBoost model it trained in-house. Where should each model run, and why would hosting both on the same platform be a mistake?
