2.4.8. Validating FM Training Data: Integrity, Safety, Cleaning, and Multimodal Bias
💡 First Principle: An FM learns from every record with equal trust. One malformed record can fail a job; a thousand near-duplicates silently overweight one pattern; a single toxic or wrong completion can be reproduced verbatim. Validation for FM data is a gate that checks format, truth, safety and balance before the expensive step runs.
1. Prompt-response pair validation (structure and meaning).
| Check | Catches |
|---|---|
| Every line parses as JSON and matches the method's schema | Truncated lines, wrong field names (output instead of completion) |
| Roles alternate correctly and every conversation ends with an assistant turn | Records with no learnable target |
| Token count per record within the model limit | Records rejected by the job or silently truncated |
| Empty, placeholder or template-residue responses removed | "TODO", "N/A", unfilled {customer_name} |
| Response actually answers its prompt (spot checks, rules, or an LLM-as-a-judge pass) | Misaligned pairs from a bad join |
| Exact and near-duplicate removal, within and across train/validation splits | Overweighted patterns and inflated validation scores |
2. Content safety screening. Screen prompts and responses before training:
- Amazon Bedrock Guardrails
ApplyGuardrailAPI evaluates any text against a configured guardrail (content filters, denied topics, word filters, sensitive information filters) without invoking a foundation model — setsourcetoINPUTfor prompts andOUTPUTfor responses, and drop or fix records where the guardrail intervenes. - Amazon Comprehend
DetectToxicContentscores English text for categories such as hate speech, harassment or abuse, insult, profanity, sexual content, and violence or threat, plus an overall toxicity score; each call takes up to 10 text segments of up to 1 KB each. - Amazon Rekognition
DetectModerationLabelsflags unsafe images in image or image-text datasets; Custom Moderation adapters adapt it to your own content.
3. Cleaning unstructured and FM data (1.3.7). The tabular techniques from 2.2.1 still apply to structured fields, and they have direct analogues for text and images:
| Tabular technique | Unstructured / FM analogue |
|---|---|
| Outlier detection (IQR, Z-score) | Flag records with extreme token length, wrong language, or embeddings far from every cluster centroid |
| Imputation | Fill missing metadata (document date, source) from reliable sources, or drop records whose missing field is required; never invent a completion |
| Exact deduplication | Hash normalized text or image bytes |
| Fuzzy deduplication (Glue FindMatches) | Near-duplicate detection by embedding similarity — S3 Vectors lists image deduplication as a use case |
4. Bias metrics across numeric, text and image data (1.3.4). The pre-training metrics from 2.3.1 — Class Imbalance (CI), Difference in Proportions of Labels (DPL), KL and Jensen-Shannon divergence — are standard formulas over facets. For numeric and categorical columns the facet is a column. For text and images the facet usually is not in the raw data at all, so you derive it: language or dialect of a text record, the demographic or capture-condition annotations of an image (skin tone band, age band, lighting, device), the region or channel in the metadata. Then compute the same metrics per facet and per modality:
- CI answers "is this group represented?" — for example, what share of face images fall in each annotated skin-tone band.
- DPL and divergence metrics answer "do labels differ by group?" — for example, whether reviews in one dialect are labeled negative far more often.
- Mitigate with the techniques in 2.3.1 and 2.4.2: targeted collection, stratified sampling, reweighting, and augmentation of under-represented groups (image transforms, entity-preserving text paraphrase), then recompute the metrics to confirm the gap closed.
Amazon SageMaker Clarify computes these pre-training metrics, but AWS notes that Clarify is no longer open to new customers (existing customers can continue to use it); the documented replacement is to compute the same standardized formulas directly with pandas and scikit-learn inside a SageMaker AI pipeline. Either way, the metric definitions are what you must know.
⚠️ Exam Trap: A dataset can be perfectly balanced on its label and still badly unbalanced on a facet. Ten thousand evenly split "defect / no defect" images all taken under factory lighting A will fail under lighting B. When a scenario describes a multimodal model that underperforms for one group or condition, the answer involves measuring representation per facet (derived from annotations or metadata) — not re-balancing the target classes.
