5.1.3. Analyzing Backlog and User Feedback
💡 First Principle: Backlog analysis and user feedback close the loop between agent performance and agent improvement. Telemetry tells you what is happening; user feedback tells you how users feel about it; backlog analysis tells you what to fix next.
Backlog Sources:
| Source | What It Contains | How to Use It |
|---|---|---|
| Unresolved conversations | Transcripts of conversations the agent couldn't resolve | Identify missing topics, knowledge gaps, and edge cases |
| Escalated conversations | Conversations transferred to humans with context | Understand where the AI capability boundary is; design for these scenarios |
| User feedback (explicit) | CSAT ratings, thumbs up/down, free-text comments | Prioritize improvements by user impact |
| User feedback (implicit) | Abandon patterns, repeat contacts, session length | Detect frustration signals users don't explicitly report |
| Agent usage patterns | Most/least used features, peak hours, channel preferences | Optimize for actual usage, not designed usage |
Continuous Improvement Cycle:
Explicit vs. Implicit Feedback, Usage and Transcripts:
- Implicit signals (abandon patterns, repeat contacts, session behavior) cover every user and move before ratings do, so they are leading indicators. Explicit feedback (CSAT, thumbs, comments) is authoritative but sparse (typically only a small share of users respond) and lagging.
- Hidden demand. If ticket volume does not fall after an agent launches with a high resolution rate (say 88%), the agent may be serving new demand from users who would never have filed a ticket, rather than deflecting existing demand. Compare deflected versus new demand before concluding the agent is not working.
- Usage can fall while metrics stay healthy (90% resolution, 4.0 CSAT). Resolution and CSAT only describe users who still engage, so ask why the others left: does the agent address the problems users actually have, do users know it exists, did an early bad experience destroy their trust, or did they move to another channel?
- Mine escalated transcripts. Cluster them into patterns, decide which the agent could have handled with better topics or knowledge, build those, test against historical samples, and watch whether escalation falls.
- Realistic targets. A single improvement cycle typically targets a modest, measurable reduction in the overall escalation rate (roughly 5-10%), not halving it. The scenario below is that size: cutting genuine-failure escalations from 50 to 30 removes 20 of 200, a 10% reduction. An agent with no feedback loop slowly becomes less relevant as needs and data change.
Prioritization Framework:
Not all issues are equal. Prioritize by combining frequency (how often does this happen?), severity (what's the user impact when it happens?), and fixability (how easy is this to resolve?). A topic that triggers incorrectly 50 times per day with a simple trigger phrase fix is higher priority than a rare edge case requiring model retraining.
Troubleshooting Scenario: A retail company's AI agent handles 5,000 conversations daily. The backlog analysis shows 200 daily escalations, but user satisfaction is 4.1 out of 5. Should the team prioritize reducing escalations? Not necessarily. Analyze the escalation categories: if 150 of 200 are intentional handoffs for complex orders (working as designed), only 50 represent genuine agent failures. The continuous improvement cycle should focus on those 50 — categorize by root cause (missing topic, incorrect answer, system error), prioritize by volume, and target the highest-impact category first. Set a measurable goal: reduce genuine failure escalations from 50 to 30 within one improvement cycle.
The key principle: raw escalation count is meaningless without categorization. Intentional handoffs are a feature, not a failure. The exam tests whether you can distinguish between healthy and unhealthy escalation patterns.
Reflection Question: An agent handles 2,000 conversations daily. Backlog analysis shows 200 escalated conversations (10%). Of these, 80 involve the same topic ("product return policy"), 60 involve questions the agent can't answer because the knowledge base is outdated, and 60 are genuine edge cases. Design the improvement plan with priorities and expected impact.
