1.1.1. The FM as a Prediction Engine
💡 First Principle: Transformer models work by computing attention scores that let every token in a sequence attend to every other token — this global attention mechanism is what allows FMs to handle long-range dependencies, follow complex instructions, and reason across multi-step chains.
Modern foundation models are built on the transformer architecture introduced in the 2017 "Attention is All You Need" paper. The critical mechanism is self-attention: when processing a token, the model learns to weight the importance of every other token in the context. This is why increasing context window size is computationally expensive (attention scales quadratically with sequence length) and why long-context models are more expensive to run.
Key concepts the exam assumes you know:
| Concept | What It Means | Why It Matters for System Design |
|---|---|---|
| Token | The base unit of text (≈ 0.75 words on average) | Cost and limits are token-based, not word-based |
| Pre-training | Initial training on massive corpus; sets base capabilities | You can't change pre-training via Bedrock — it's baked in |
| Inference | Running the trained model to generate output | What Bedrock does — you pay per token of input + output |
| Parameters | The model's learned weights (billions to trillions) | More params ≠ better for all tasks; see Domain 2 |
| Context window | The total token limit for input + output combined | Determines what the model can "see" in one call |
| Temperature | Controls randomness of token selection | Higher = more creative/varied; lower = deterministic |
| Training cutoff | Pre-training data ends at a fixed date; the model does not update itself afterwards | Post-cutoff or changed facts must come from retrieval (RAG), not from the model |
| Generalization | Patterns learned in pre-training transfer to tasks the model was never explicitly trained on (zero-/few-shot) | One FM serves many tasks through prompting alone — no task-specific rules, lookup tables or web access involved |
Autoregressive generation: FMs generate output one token at a time, with each new token conditioned on all previous tokens. This is why streaming responses become possible (you can emit tokens as they're generated) and why truncating output mid-stream creates inconsistent results. Each token is chosen because it is statistically plausible, not because it was checked against a source — so a fluent, specific but wrong detail (a date, a figure) is normal model behavior, not a failure of attention or of the temperature setting.
⚠️ Exam Trap: Candidates often confuse context window limits with memory. The context window resets every API call unless you explicitly pass conversation history. "Memory" in an agent or chatbot is always implemented externally — in DynamoDB, in Bedrock Knowledge Bases session context, or in custom stores.
Position effects ("lost in the middle"): fitting in the context window is not the same as being used. In very long inputs, models attend most reliably to content near the beginning and the end; clauses buried in the middle are missed more often. Put critical instructions at the start or end, and process long documents in sections (map-reduce summarization) rather than as one block.
Reflection Question: If a foundation model generates text by predicting statistically likely next tokens rather than retrieving facts, what architectural component must a production system add to ensure factually accurate responses about proprietary company data?
