2.1.2. Evaluating Data Quality for Grounding

Every AI agent's effectiveness is bounded by the quality of its grounding data. The exam tests five specific data quality dimensions — and expects you to know which dimension is likely the root cause when an agent underperforms.

Phase 1 introduced these five factors (accuracy, relevance, timeliness, cleanliness, availability). Here, we go deeper into how to evaluate each during the planning phase:

Data Assessment Checklist for Architects:

1. Inventory existing data sources — List every source the agent will need. For each, document the owner, update frequency, access method, and known quality issues.
2. Map data to agent scenarios — For each agent use case, identify which data sources feed which responses. Gaps here mean the agent will hallucinate on those topics.
3. Test retrieval quality — Before building the agent, run sample queries against the knowledge base. If retrieval returns irrelevant or incomplete results, fix the data pipeline before proceeding.
4. Define freshness requirements — For each data source, agree on the maximum acceptable lag. Real-time CRM data and monthly policy documents have very different requirements.
5. Plan for data governance — Who approves changes to grounding data? How are errors in grounding data reported and corrected?

Diagnosing common grounding failures:

• Right content updated, old answers persist — check that the update reached the agent's knowledge source (sync or refresh), not just the source system, and that no authored topic or agent instruction still contains the old wording. Retraining the model is not the fix: grounding is retrieval, not model weights.
• Old and new versions of a document both indexed — retrieval is ambiguous. Remove or archive the outdated version and refresh the index; a prompt telling the agent to prefer newer documents is a workaround, not a fix.
• Authoritative sources disagree (for example, WHO versus a national authority) — do not let the agent merge them. Keep source attribution, detect the conflict and present each recommendation separately with its source.
• Retrieval returns relevant documents but answers ignore their content — look at document structure: tables, images, scanned pages and complex layouts often cannot be extracted. Restructure the content into extractable text.
• One source is far less available than the others — a legacy on-premises database depends on local infrastructure (maintenance windows, network, hardware) and usually has lower uptime than cloud services such as Dataverse or SharePoint. Mitigate with a caching layer or scheduled replication to cloud storage so the agent can still ground when the source is down.
• What blocks deployment — widespread duplicate or contradictory content (it changes answers). Tolerable for launch: a small coverage gap on a low-volume topic, a minor latency miss, or a small share of aged evergreen content.

Exam Trap: The exam may present a scenario where an agent "works well in testing but fails in production." The most common cause is data quality degradation — test environments use clean, curated data while production deals with messy, incomplete, real-world data. Always evaluate data quality against production conditions, not test conditions.

Reflection Question: An HR agent correctly answers questions about vacation policy for US employees but gives wrong answers for UK employees. The policy documents for both regions are uploaded. What data quality factor would you investigate first?