2.4.7. Preparing Data for Fine-Tuning, Continued Pre-Training, and Distillation
💡 First Principle: Each customization method learns something different, so each needs a different shape of data. Fine-tuning learns a task from labeled examples, continued pre-training absorbs a domain from unlabeled text, and distillation copies a large model's behaviour into a small one from prompts. Pick the method first; the data format follows.
All three Amazon Bedrock methods use JSON Lines (.jsonl) files in S3 — one JSON object per line — but the object shape differs:
| Method | What the data teaches | Record shape | Labeling needed |
|---|---|---|---|
| Fine-tuning (non-conversational models) | A task: input → desired output | {"prompt": "...", "completion": "..."} | Yes — every record needs the ideal output |
| Fine-tuning (Converse-format models) | A conversational behaviour | {"schemaVersion": "bedrock-conversation-2024", "system": [...], "messages": [user, assistant, ...]} | Yes — assistant turns are the targets |
| Continued pre-training | Domain vocabulary and knowledge | {"input": "..."} | No — unlabeled domain text |
| Distillation | A teacher model's responses to your use-case prompts | Prompts in the conversation schema; optional prompt-response pairs as golden examples | No — the teacher generates responses |
Fine-tuning data preparation:
- Match the schema to the model. Converse-API models expect
system,userandassistantmessages with alternating roles; image-text models reference images by S3 URI inside the message content. - Respect per-model limits. For example, Meta Llama 3.1 requires 100 to 10,000 training plus validation records (adjustable) and up to 16,000 input plus output tokens per record; Anthropic Claude 3 Haiku requires at least 32 records. Estimate tokens at about 6 characters per token for planning, then verify with the target model.
- Quality over volume. Completions must be exactly the output you want in production — tone, format, length. Inconsistent completions teach inconsistency.
- Hold out validation data from the same distribution, with no overlap with the training records (deduplicate across splits, not only within them).
Continued pre-training preparation is closer to corpus building: gather representative unlabeled domain text (manuals, filings, research), remove boilerplate and duplicates, mask sensitive values, and split long documents so each input fits the model's token limit.
Distillation preparation. Bedrock Model Distillation takes your prompts, generates responses from the teacher model, can apply data synthesis (for example generating similar prompts) to improve diversity, and fine-tunes the student. Two input paths exist:
- Your own prompts in JSONL, optionally with a handful of labeled prompt-response pairs that Bedrock uses as golden examples.
- Invocation logs from production traffic: either the prompts only (Bedrock regenerates teacher responses) or prompt-response pairs (Bedrock reuses the logged responses — the teacher model in the job must match the model that produced the logs). Add
requestMetadatawhen invoking the model so the job can filter logs to one use case.
You can supply at most 15,000 prompts or prompt-response pairs, and data synthesis can expand the fine-tuning set to at most 15,000 prompt-response pairs while incurring teacher-model inference charges; reusing logged prompt-response pairs avoids regenerating teacher responses.
⚠️ Exam Trap: Continued pre-training is the answer when the organization has lots of unlabeled domain text and the model lacks domain vocabulary; fine-tuning is the answer when it has labeled input-output examples of a task. A stem offering "thousands of internal research papers, no labels" points to continued pre-training; "5,000 support tickets with the approved agent reply" points to fine-tuning; "a large model already performs well but is too slow or costly" points to distillation.
