5.1.3. Analyzing Backlog and User Feedback
💡 First Principle: Backlog analysis and user feedback close the loop between agent performance and agent improvement. Telemetry tells you what is happening; user feedback tells you how users feel about it; backlog analysis tells you what to fix next.
Backlog Sources:
| Source | What It Contains | How to Use It |
|---|---|---|
| Unresolved conversations | Transcripts of conversations the agent couldn't resolve | Identify missing topics, knowledge gaps, and edge cases |
| Escalated conversations | Conversations transferred to humans with context | Understand where the AI capability boundary is; design for these scenarios |
| User feedback (explicit) | CSAT ratings, thumbs up/down, free-text comments | Prioritize improvements by user impact |
| User feedback (implicit) | Abandon patterns, repeat contacts, session length | Detect frustration signals users don't explicitly report |
| Agent usage patterns | Most/least used features, peak hours, channel preferences | Optimize for actual usage, not designed usage |
Continuous Improvement Cycle:
Prioritization Framework:
Not all issues are equal. Prioritize by combining frequency (how often does this happen?), severity (what's the user impact when it happens?), and fixability (how easy is this to resolve?). A topic that triggers incorrectly 50 times per day with a simple trigger phrase fix is higher priority than a rare edge case requiring model retraining.
Troubleshooting Scenario: A retail company's AI agent handles 5,000 conversations daily. The backlog analysis shows 200 daily escalations, but user satisfaction is 4.1 out of 5. Should the team prioritize reducing escalations? Not necessarily. Analyze the escalation categories: if 150 of 200 are intentional handoffs for complex orders (working as designed), only 50 represent genuine agent failures. The continuous improvement cycle should focus on those 50 — categorize by root cause (missing topic, incorrect answer, system error), prioritize by volume, and target the highest-impact category first. Set a measurable goal: reduce genuine failure escalations from 50 to 30 within one improvement cycle.
The key principle: raw escalation count is meaningless without categorization. Intentional handoffs are a feature, not a failure. The exam tests whether you can distinguish between healthy and unhealthy escalation patterns.
Reflection Question: An agent handles 2,000 conversations daily. Backlog analysis shows 200 escalated conversations (10%). Of these, 80 involve the same topic ("product return policy"), 60 involve questions the agent can't answer because the knowledge base is outdated, and 60 are genuine edge cases. Design the improvement plan with priorities and expected impact.
