2.3.3. Model Routing for Cost and Performance Optimization
Model routing is an advanced architectural pattern that intelligently directs AI requests to the most suitable model based on task complexity, cost, and latency requirements. Instead of sending every request to the most powerful (and expensive) model, a model router evaluates the request and selects the optimal model.
This is a distinct syllabus bullet — "Implement a model router to intelligently route requests to the most suitable model" — indicating the exam expects specific knowledge about this pattern.
Why Model Routing Matters:
Without routing, organizations face a trade-off: use a powerful model for everything (expensive, slow) or use a cheap model for everything (fast, inaccurate on complex tasks). Model routing eliminates this trade-off by matching task complexity to model capability.
How Model Routing Works:
Routing Criteria:
| Criterion | Description | Example |
|---|---|---|
| Task complexity | Simple lookups vs. multi-step reasoning | FAQ → SLM; Legal analysis → LLM |
| Cost budget | Per-request cost ceiling | Internal tools tolerate higher latency for lower cost |
| Latency requirement | Response time SLA | Customer-facing chat needs <2s; batch processing tolerates minutes |
| Accuracy requirement | Acceptable error rate | Financial calculations need high accuracy; draft suggestions can be approximate |
| Data sensitivity | Classification level of input data | Confidential data may require models with specific data residency guarantees |
| Capability / modality | Whether the request needs a capability only some models have | Text-only requests → text model; requests containing an image → multimodal model |
Implementation in Microsoft Foundry:
Microsoft Foundry supports model routing through its model catalog and deployment options. Architects can configure:
- Multiple model deployments — Deploy different models (GPT-4o, Phi-4, custom models) as separate endpoints
- Routing logic — Use agent orchestration or API gateway logic to evaluate requests and route to appropriate endpoints
- Fallback chains — If the primary model is unavailable or overloaded, route to a secondary model
- Cost monitoring — Track per-model costs to optimize routing rules over time
Operating a model router:
- Safety before cost — run a safety or urgency classifier before the cost-based router and force flagged queries (for example, urgent patient symptoms) to the most capable model or a human path; cost routing alone can send a high-risk query to a weak tier.
- Latency — the classification step adds a small overhead (typically tens of milliseconds for a lightweight classifier), but routing simple queries to faster models usually lowers net latency for them.
- Monitoring — track accuracy, latency and cost per tier, plus the classifier's own accuracy, continuously rather than only at launch. Per-tier views catch a tier whose accuracy is dropping, classifier drift (complex queries sent to the cheap tier) and cost anomalies (simple queries sent to the expensive tier); aggregated or most-expensive-tier-only views hide all three.
- Drift — if one tier degrades while the others stay stable, suspect that query patterns shifted and new query types are being misclassified as simple; review the classifier and thresholds.
Cost Optimization Impact:
A well-implemented model router can reduce inference costs by 40-70% compared to using a single high-capability model for all requests. The savings come from routing the majority of simple requests (which typically comprise 60-80% of volume) to cheaper, faster models.
Exam Trap: Model routing is NOT load balancing. Load balancing distributes requests evenly across identical instances of the same model. Model routing sends different requests to different models based on request characteristics. The exam may present these as equivalent — they are not.
Reflection Question: An enterprise deploys an AI agent that handles 10,000 requests per day. Analysis shows 70% are simple FAQ lookups, 20% require moderate reasoning, and 10% need complex multi-step analysis. Design a model routing strategy and estimate the cost impact compared to using a single LLM.
