2.4. Preparing Data for Foundation Models and RAG

💡 First Principle: A foundation model never sees your raw files. It sees tokens during training and retrieved chunks at inference time, and both are artifacts your data pipeline produces. Whatever you get wrong upstream — a scanned table the parser flattened, a chunk cut mid-sentence, a customer's phone number left in a fine-tuning record — is faithfully learned or faithfully retrieved. Data preparation for FMs is the same discipline as the tabular work in 2.1–2.3, applied to a different unit of data.

Sections 2.1 through 2.3 prepare features: numbers and categories for a model you train from scratch. This section prepares data for models someone else pre-trained. The questions change shape. Instead of "how do I scale this column?" you ask "how do I split this 80-page manual so the right paragraph is retrievable?", "which embedding model puts a product photo and its description in the same vector space?", and "how do I prove that none of these 20,000 prompt-response pairs leaks a customer's account number?"

Think of it as two pipelines that share a front end. The retrieval pipeline (RAG) turns documents, images and audio into embeddings stored in a vector database, so a model can look things up at query time. The customization pipeline turns curated examples into JSONL training files for fine-tuning, continued pre-training or distillation, so the model's weights change. Both need the same upstream hygiene: parse and ingest the modality correctly, remove or mask sensitive data, deduplicate, and check that the data is safe and representative.

⚠️ Common Misconception: "RAG means I don't need data preparation — the model reads the documents." The model reads only the chunks the retriever returns, and the retriever can return only what was parsed, chunked and embedded well. Most poor RAG answers trace back to ingestion (a table parsed as a run-on line, a chunk too large to rank well, missing metadata for filtering), not to the generating model.