2.1.2. Evaluating Data Quality for Grounding
Every AI agent's effectiveness is bounded by the quality of its grounding data. The exam tests five specific data quality dimensions — and expects you to know which dimension is likely the root cause when an agent underperforms.
Phase 1 introduced these five factors (accuracy, relevance, timeliness, cleanliness, availability). Here, we go deeper into how to evaluate each during the planning phase:
| Factor | Assessment Questions | Red Flags |
|---|---|---|
| Accuracy | Is the data verified? Who maintains it? When was it last audited? | No data owner assigned; no validation process; data imported from unknown sources |
| Relevance | Does the data contain the information agents need to answer user questions? | Data exists but doesn't cover the domain; high percentage of irrelevant results in test queries |
| Timeliness | How frequently is the data updated? What's the acceptable lag? | Batch updates (daily/weekly) for data that changes hourly; no refresh pipeline defined |
| Cleanliness | Is data deduplicated? Are formats consistent? Are there gaps? | Multiple records for the same entity; inconsistent naming conventions; null values in key fields |
| Availability | Can agents access the data at inference time? What are the dependencies? | Data behind VPN with no agent connectivity; single point of failure in data pipeline; no SLA on data source uptime |
Data Assessment Checklist for Architects:
- Inventory existing data sources — List every source the agent will need. For each, document the owner, update frequency, access method, and known quality issues.
- Map data to agent scenarios — For each agent use case, identify which data sources feed which responses. Gaps here mean the agent will hallucinate on those topics.
- Test retrieval quality — Before building the agent, run sample queries against the knowledge base. If retrieval returns irrelevant or incomplete results, fix the data pipeline before proceeding.
- Define freshness requirements — For each data source, agree on the maximum acceptable lag. Real-time CRM data and monthly policy documents have very different requirements.
- Plan for data governance — Who approves changes to grounding data? How are errors in grounding data reported and corrected?
Exam Trap: The exam may present a scenario where an agent "works well in testing but fails in production." The most common cause is data quality degradation — test environments use clean, curated data while production deals with messy, incomplete, real-world data. Always evaluate data quality against production conditions, not test conditions.
Reflection Question: An HR agent correctly answers questions about vacation policy for US employees but gives wrong answers for UK employees. The policy documents for both regions are uploaded. What data quality factor would you investigate first?
