2.3.3. Model Routing for Cost and Performance Optimization
Model routing is an advanced architectural pattern that intelligently directs AI requests to the most suitable model based on task complexity, cost, and latency requirements. Instead of sending every request to the most powerful (and expensive) model, a model router evaluates the request and selects the optimal model.
This is a distinct syllabus bullet — "Implement a model router to intelligently route requests to the most suitable model" — indicating the exam expects specific knowledge about this pattern.
Why Model Routing Matters:
Without routing, organizations face a trade-off: use a powerful model for everything (expensive, slow) or use a cheap model for everything (fast, inaccurate on complex tasks). Model routing eliminates this trade-off by matching task complexity to model capability.
How Model Routing Works:
Routing Criteria:
| Criterion | Description | Example |
|---|---|---|
| Task complexity | Simple lookups vs. multi-step reasoning | FAQ → SLM; Legal analysis → LLM |
| Cost budget | Per-request cost ceiling | Internal tools tolerate higher latency for lower cost |
| Latency requirement | Response time SLA | Customer-facing chat needs <2s; batch processing tolerates minutes |
| Accuracy requirement | Acceptable error rate | Financial calculations need high accuracy; draft suggestions can be approximate |
| Data sensitivity | Classification level of input data | Confidential data may require models with specific data residency guarantees |
Implementation in Microsoft Foundry:
Microsoft Foundry supports model routing through its model catalog and deployment options. Architects can configure:
- Multiple model deployments — Deploy different models (GPT-4o, Phi-4, custom models) as separate endpoints
- Routing logic — Use agent orchestration or API gateway logic to evaluate requests and route to appropriate endpoints
- Fallback chains — If the primary model is unavailable or overloaded, route to a secondary model
- Cost monitoring — Track per-model costs to optimize routing rules over time
Cost Optimization Impact:
A well-implemented model router can reduce inference costs by 40-70% compared to using a single high-capability model for all requests. The savings come from routing the majority of simple requests (which typically comprise 60-80% of volume) to cheaper, faster models.
Exam Trap: Model routing is NOT load balancing. Load balancing distributes requests evenly across identical instances of the same model. Model routing sends different requests to different models based on request characteristics. The exam may present these as equivalent — they are not.
Reflection Question: An enterprise deploys an AI agent that handles 10,000 requests per day. Analysis shows 70% are simple FAQ lookups, 20% require moderate reasoning, and 10% need complex multi-step analysis. Design a model routing strategy and estimate the cost impact compared to using a single LLM.
