3.2.2. Establishing Baselines Before AI

💡 First Principle: Improvement is a difference between two numbers, and you cannot compute a difference if you never recorded the first number. Once AI is deployed, the "before" state is gone forever. Baselines must be captured before implementation, using the same definitions you will use afterward.

What to baseline depends on the KPIs, but typically includes cycle time, cost per unit of work, volume handled per person, error or rework rate, customer satisfaction, and compliance or quality measures. Capture enough history to account for normal variation: a single week can mislead if it falls during a seasonal peak.

Attribution is the hard part. Many things change at once: a new product launches, staffing changes, a marketing campaign runs. To separate AI's effect from everything else, strategists use:

• Control groups: one team, region, or customer segment uses the AI solution while a comparable one does not, for the same period.
• A/B testing: randomly assign cases or users to the AI-assisted and current process and compare outcomes.
• Phased rollouts: stagger deployment so earlier and later groups can be compared.
• Before-and-after with adjustment: when controls are impossible, compare before and after while documenting and adjusting for known external changes.

Baselines also protect against measurement drift, where the definition of a metric quietly changes after launch ("resolved" starts to include cases transferred to another team). Agree on definitions, data sources, and measurement method before go-live, and document them.

⚠️ Exam Trap: When a scenario describes an organization that launched AI and now cannot prove its impact, the root cause is usually a missing pre-implementation baseline. When asked what to do before deploying an AI solution to ensure value can be demonstrated, establishing baseline metrics (with consistent definitions and ideally a comparison group) is the best answer.

Reflection Question: A bank plans to roll out an AI assistant to 1,200 branch staff next month. Design a baseline and attribution approach that would let the bank credibly report the assistant's effect on time per customer request six months later.