3.4. Foundation Model Selection, Customization and Evaluation
💡 First Principle: A foundation model is a component you configure, not a model you train. The engineering work moves from choosing algorithms and hyperparameters to choosing a model, deciding how much of its behaviour to change (prompt, retrieval, or weights), and proving — with evidence you can reproduce — that the result is good enough. Every decision in this section is a point on one ladder: change the cheapest, most reversible thing first, and only move up when measurement says you must.
What fails when this discipline is missing? A team picks the largest model on the leaderboard, discovers its latency breaks the chat experience, fine-tunes a smaller model to recover quality, and then cannot say whether the fine-tuned model is better than the original — because nobody fixed the prompt, the test set or the scoring method before starting. Three expensive steps, no defensible answer. The same team with a saved prompt version, an evaluation job over a fixed dataset, and a tracked experiment would have learned in a day that a better prompt plus retrieval closed the gap without touching weights.
Think of it like staffing a help desk. You can hire a generalist (pick a model), hand them a well-written procedures binder (prompt engineering), give them access to the filing cabinet (RAG), or send them on a months-long training course (fine-tuning). The binder and the filing cabinet are cheap and you can change them tomorrow; the course is expensive and hard to undo. And whichever you choose, you need a consistent way to grade their answers.
⚠️ Common Misconception: "Fine-tuning is how you teach a model your company's facts." Fine-tuning changes behaviour — style, format, task skill, domain vocabulary. Facts that change, or that must be cited, belong in retrieval. A model fine-tuned on last quarter's price list will confidently quote last quarter's prices; a RAG system reads the current list at query time and can point to the source.
