3.4.2. Customizing Foundation Models: Prompting, Fine-Tuning, Distillation and Reinforcement Fine-Tuning
💡 First Principle: Customization techniques differ in what they change and what data they need. Prompt engineering changes only the input. Retrieval changes only what the model reads. Fine-tuning variants change the model's weights, and each needs a different kind of data. Match the technique to the gap you measured, and to the data you actually have.
Task-specific prompt engineering is the first lever because it is free to change and instantly reversible:
| Technique | What it does | Use when |
|---|---|---|
| Clear instruction + context | States the task, audience, constraints and input | Always — most "model is bad" findings are underspecified prompts |
| Few-shot examples | Includes input/output pairs in the prompt (in-context learning) | Output format or labeling scheme must be consistent |
| Chain-of-thought / step-by-step | Asks the model to reason before answering | Multi-step reasoning, math, policy application |
| Prompt templates with variables | Reusable prompt with placeholders filled at runtime | The same task runs across many inputs and teams |
| Inference parameters | Temperature, top P, top K, maximum length, stop sequences | Lower temperature for deterministic extraction; higher for creative variety |
Temperature reshapes the next-token probability distribution: a low value makes high-probability tokens more likely (more deterministic output), a high value flattens the distribution (more varied output). Top P limits sampling to the top share of cumulative probability; top K limits it to the K most likely tokens.
Foundation model customization methods. Amazon Bedrock currently offers supervised fine-tuning, distillation and reinforcement fine-tuning; each produces a private custom model, and training is billed on tokens processed × epochs plus monthly model storage. Continued pre-training on unlabeled text is done on SageMaker AI — for example JumpStart domain adaptation fine-tuning (below) — rather than as a Bedrock customization job.
| Method | Where | Data you provide | What it changes | Choose when |
|---|---|---|---|---|
| Supervised fine-tuning | Bedrock, SageMaker AI | Labeled prompt–completion pairs (JSONL), or Converse-format conversations | Task behaviour, format, tone, domain vocabulary | You have hundreds to thousands of high-quality examples of the exact output you want |
| Continued pre-training (domain adaptation) | SageMaker AI (JumpStart domain adaptation) | Unlabeled domain text (a CSV, JSON or TXT file) | Domain knowledge and language familiarity | You have a large corpus of domain text but no labeled examples |
| Distillation | Bedrock | Prompts (optionally some labeled pairs), or existing invocation logs | Transfers a larger teacher model's quality to a smaller, faster, cheaper student | A big model meets the quality bar but is too slow or too expensive in production |
| Reinforcement fine-tuning | Bedrock (selected models) | Prompts plus a reward function (Lambda code or a model-as-a-judge grader); invocation logs can supply prompts | Optimizes toward a measurable definition of a good response | Quality is gradable but labeled "gold" answers are expensive or impossible to write |
SageMaker AI JumpStart fine-tuning covers the open-weight models you host yourself: domain adaptation (unlabeled domain text — continued pre-training) or instruction-based fine-tuning (prompt–response examples), with hyperparameters such as epoch, learning_rate, per_device_train_batch_size and LoRA settings (lora_r, lora_alpha) for parameter-efficient fine-tuning.
Fine-tuning hygiene carries over from 3.2: hold out a validation set, watch for overfitting (training loss falling while validation loss rises), and guard against catastrophic forgetting with a low learning rate, few epochs, or parameter-efficient methods that leave most weights frozen.
After customization: a custom Bedrock model is served either through Provisioned Throughput (dedicated capacity, consistent performance) or by deploying it for on-demand inference (pay per use, no provisioned capacity) where supported. Models customized outside Bedrock — for example open-weight models fine-tuned on SageMaker AI — can be brought in with Amazon Bedrock Custom Model Import (Hugging Face safetensors weights from Amazon S3, supported architectures only).
⚠️ Exam Trap: Distillation needs prompts, not labeled answers — Bedrock generates the teacher's responses for you (or reuses teacher responses already in invocation logs, provided the teacher model matches). If a scenario says "we already log production prompts and responses from the large model and want a cheaper model," distillation from invocation logs is the signal.
⚠️ Exam Trap: "No labeled data, but we can write code that scores whether an answer is right" points to reinforcement fine-tuning, not supervised fine-tuning. "Lots of unlabeled domain documents" points to continued pre-training (domain adaptation fine-tuning on SageMaker AI JumpStart). "Answers must reflect documents that change weekly" points to RAG, not any fine-tuning method.
