1.1.1. The Anatomy of a Production AI Solution
💡 First Principle: Every production AI application decomposes into the same five layers — entry point, orchestration, model access, data/context, and cross-cutting concerns — because every AI request follows the same life cycle: receive, enrich with context, call the model, persist, and observe. Learn the layers once and you can place any Azure service into its slot on sight.
Picture a customer-support copilot in production. A user's question arrives at an API. The API can't just forward the question to a language model — the model knows nothing about this company's refund policy. So an orchestration layer first retrieves relevant documents, assembles them into a prompt, calls the model endpoint, stores the conversation, and emits telemetry. If the request involves heavy work (summarizing a 200-page PDF), the API drops a message on a queue and a background worker picks it up, so the user isn't left staring at a spinner.
The flow looks like this:
Notice what this diagram implies. The model box is one node — everything else is standard cloud engineering, and everything else is what AI-200 tests. The vector store gives the model context (Phase 2). The entry point and worker run in containers (Phase 3). The queue decouples slow work from fast responses (Phase 4). Secrets, config, and telemetry wrap the whole thing (Phase 5). The four exam domains are literally the four regions of this diagram.
What breaks without this structure? Remove the queue and your API times out whenever the model is slow. Remove the vector store and the model hallucinates answers about your refund policy. Remove telemetry and a latency spike somewhere between five services becomes unfindable. The exam's scenario questions are almost all "something is missing or misconfigured in this anatomy — fix it."
⚠️ Exam Trap: When a scenario mentions long-running or bursty AI workloads and asks how to keep the front end responsive, the answer is asynchronous decoupling (queue + worker), not a bigger compute SKU. Scaling up treats the symptom; decoupling fixes the architecture. Watch for distractors offering "increase the App Service plan tier."
Reflection Question: A teammate proposes calling the model endpoint directly from the HTTP request handler and scaling out instances to handle load. Using the anatomy, what two problems will surface first, and which layers are missing?
