3.4.4. Performance, Latency and Cost Trade-offs for AI Models
💡 First Principle: For a foundation model, cost and latency are driven by tokens and by model size. Every lever either sends fewer tokens, processes the same tokens on a smaller model, reuses work already done, or trades immediacy for a discount. Name which of those the scenario needs and the answer follows.
| Lever | What it trades | Choose when |
|---|---|---|
| Smaller model / distilled model | Some capability for speed and price | A smaller model passes your evaluation, or can be distilled to |
| Intelligent prompt routing | Routes each request between two models in the same family based on predicted response quality | Traffic mixes easy and hard prompts; you want quality where needed, cost savings elsewhere |
| Prompt caching | Reuses a static prompt prefix (system prompt, long document, tool definitions); cache reads are billed at a reduced rate | Long, repeated context across many requests; place static content first |
| Batch inference | Asynchronous processing from S3 at a lower price than on-demand for supported models | Large offline jobs with no user waiting |
| Service tiers | Priority (fastest, premium price), Standard (default), Flex (discount, longer processing), Reserved (reserved tokens-per-minute capacity) | Match urgency to price per request |
| Latency-optimized inference | performanceConfig latency set to optimized on supported models (preview) | Interactive apps on a supported model where time matters most |
| Provisioned Throughput | Purchased dedicated capacity (commitment terms available) | Steady high volume, or serving a custom model with guaranteed throughput |
| Cross-Region inference | Routes requests across Regions for throughput; geographic profiles keep data in a geography, global profiles cost less | Throttling at peak; choose geographic when residency rules apply |
| Fewer tokens | Shorter prompts, fewer retrieved chunks (with reranking), max tokens limits | Output length or bloated context drives cost |
How to decide: pick the quality bar first (from an evaluation), then the latency budget (time to first token for chat, total time for batch), then minimize cost among options that satisfy both. A model that is 20% cheaper but fails the quality bar is not cheaper — it moves the cost into support tickets.
⚠️ Exam Trap: Batch inference and the Flex tier save money by giving up immediacy — both are wrong for a customer-facing chatbot with a latency SLA. Conversely, paying for the Priority tier or Provisioned Throughput for a nightly summarization job wastes money.
⚠️ Exam Trap: Prompt caching helps only when the prefix is identical across requests. Putting the user's question before the long static document defeats the cache; static content goes first, variable content last.
