2.5. Evaluating and Customizing Generative Solutions

💡 First Principle: A generative system isn't done when it runs — it's done when it's measured. Evaluation scores output quality dimensions (groundedness, relevance, coherence, safety, plus agent-specific tool metrics), usually with model-graded evaluators, and customization (fine-tuning) adjusts the model's defaults only when prompting and grounding aren't enough.

Why care: GenAIOps and evaluation are woven through both the gen-AI and planning domains, and the exam's production-readiness framing means "how do you know it's working?" is a recurring theme. Evaluation is the difference between "it answered" and "it answered correctly, on-source, safely, and used its tools well."

⚠️ Common Misconception: "Evaluating a generative app means checking it doesn't crash." Functional pass/fail misses the whole point — a generative app can run flawlessly while being ungrounded, irrelevant, or unsafe. Evaluation measures quality, not just liveness.