2.4.2. Tokenization and Domain-Specific Text Augmentation
💡 First Principle: A model's budget is measured in tokens, not characters or words. Context windows, chunk sizes, embedding input limits, fine-tuning record limits and inference cost are all token counts — and each model family tokenizes differently. Planning text data without counting tokens is like packing for a flight without knowing the bag's weight limit.
2.2.3 introduced tokenization as an encoding step for classic NLP. For FMs it matters for a second reason: capacity planning. Modern models use subword tokenizers (byte-pair encoding and similar), so common words are single tokens while rare domain terms, product codes and non-English text split into several. A chemistry corpus or a codebase therefore consumes more tokens per character than general English prose.
| Where tokens constrain you | Verified example |
|---|---|
| Embedding input | Titan Text Embeddings V2 accepts up to 8,192 tokens or 50,000 characters; Cohere Embed v3 accepts 512 tokens (about 2,048 characters) per text |
| Chunk size | Bedrock KB default chunking targets about 300 tokens; fixed-size chunking is configured in maxTokens |
| Fine-tuning records | Each model sets a per-record token ceiling (Meta Llama 3.1: 16,000 input plus output tokens) |
| Characters-per-token rules of thumb | Titan embeddings documentation: about 4.7 characters per token in English; Bedrock customization guidance: about 6 characters per token for dataset planning |
Because rules of thumb differ by model, do not reuse one model's ratio for another. The Bedrock CountTokens API returns the exact input token count a given model would charge for a request, without running inference and without charge — use it to validate chunk sizes and training records against the model you will actually use.
Text normalization before tokenization. Standardize Unicode, strip markup and boilerplate (navigation text, repeated headers and footers, legal disclaimers on every page), and fix encoding errors. Boilerplate is especially costly in RAG: repeated footers produce many near-identical chunks that crowd real content out of the top-k results.
Domain-specific augmentation. When a specialized corpus is small or unbalanced, augment it in ways that preserve meaning in that domain:
| Technique | How it works | Domain risk to control |
|---|---|---|
| Domain-dictionary synonym swap | Replace terms with approved synonyms or abbreviations ("myocardial infarction" / "MI") | Generic thesauri swap terms that are not equivalent in the domain |
| Back-translation | Translate to another language and back (for example with Amazon Translate) to paraphrase | Technical terms and numbers can drift; verify key entities survive |
| Template filling | Generate variations from templates with real entity lists (product names, error codes) | Templates create repetitive phrasing if overused |
| LLM-generated synthetic examples | Prompt a stronger FM to produce new question-answer pairs grounded in source text | Hallucinated facts; must be validated before training (see 2.4.8) |
| Noise injection | Add realistic typos or OCR-style errors | Only useful when production inputs contain the same noise |
⚠️ Exam Trap: Augmentation that changes the label-relevant content is corruption, not augmentation. Swapping "benign" for "malignant" as synonyms, or back-translating a dosage and losing a decimal, teaches the model something false. The exam favours domain-aware augmentation (curated dictionaries, entity-preserving paraphrase, validated synthetic data) over generic augmentation when the corpus is specialized.
