3.2.1.4. Context and Token Management

3.2.1.4. Context and Token Management

Every model has a context window limit—the maximum tokens (input + output combined) it can process. GPT-4o supports 128K tokens; GPT-3.5-Turbo only 16K. Exceeding this limit causes API errors.

What breaks without context management:

• Long conversations fail with "context length exceeded" errors
• Important early context gets truncated in naive implementations
• Costs explode when you send unnecessary context every request

Strategies for managing context:

• Summarize older conversation turns instead of including full history
• Use embeddings to retrieve only relevant context (RAG pattern)
• Track token counts with tiktoken library before sending requests

Azure OpenAI Chat Completions