2.3.2. Tokens and Context Windows

💡 First Principle: Models read and write in tokens (small chunks of text, roughly three-quarters of an English word on average), and every model can consider only a limited number of tokens at once: its context window. Everything the model needs for a request (instructions, conversation history, retrieved documents, and its own answer) must fit in that window, and every token is billed and takes time to process. Context is a budget, not a free resource.

Business consequences of token limits appear in four ways:

• Truncation and forgetting. If input exceeds the window, something is cut, often the earliest part of a long conversation or the end of a long document. A chatbot that "forgets" what the customer said twenty messages ago has usually hit its context limit.
• Output limits. Models also cap how many tokens they generate per response. A request to "summarize all 400 contracts in one answer" may stop mid-sentence.
• Cost. Consumption-based pricing (for example, Amazon Bedrock on-demand pricing) charges per input and output token. Sending a 100-page manual with every question multiplies cost by every request. Long contexts are a major driver of generative AI spend.
• Latency and attention. Longer inputs take longer to process, and models can overlook details buried in the middle of very long contexts. More context can make answers worse if the relevant fact is hidden among irrelevant text.

Practical strategies that a strategist should expect teams to use:

• Retrieve, don't dump: use RAG to send only the most relevant passages for each question.
• Chunk and summarize: split long documents, summarize sections, then work with summaries.
• Manage conversation memory: summarize earlier conversation turns instead of resending them in full.
• Right-size the model: choose a model whose context window and price fit the task; the largest window is not always necessary.
• Cache repeated context: some services (including Amazon Bedrock) offer prompt caching that reduces the cost of resending the same long instructions.

⚠️ Exam Trap: When a scenario describes a generative AI system that ignores earlier information, cuts off long answers, or becomes slow and expensive as documents grow, think context window and token limits. The better answer usually narrows or restructures what is sent (retrieval, chunking, summarization) rather than simply choosing a larger model or pasting in more text.

Reflection Question: An HR assistant answers questions by sending the entire 300-page employee handbook with every query. Costs are high and answers sometimes miss relevant policies. Explain both problems in terms of tokens and context windows, and propose a better design at a business level.