6.1.1. Prompt Caching and Token Optimization
💡 First Principle: Prompt caching eliminates the cost of re-processing static content — system prompts, few-shot examples, and large knowledge base extracts that are identical across many requests — by computing their token representations once and reusing the cached result for subsequent calls.
Amazon Bedrock prompt caching mechanics:
The Bedrock API supports prompt caching for content prefixes that are identical across calls. When the same system prompt (or document context) appears at the start of a request body, Bedrock detects the cache hit and skips re-processing those tokens:
# Prompt caching via cache_point markers in Converse API
response = bedrock_runtime.converse(
modelId='anthropic.claude-3-sonnet-20240229-v1:0',
system=[
{'text': LONG_SYSTEM_PROMPT}, # 2,000 tokens — same every request
{'cachePoint': {'type': 'default'}} # Mark end of cacheable prefix
],
messages=[
{
'role': 'user',
'content': [
{'text': LARGE_DOCUMENT_CONTEXT}, # 8,000 tokens of retrieved docs
{'cachePoint': {'type': 'default'}},
{'text': user_query} # Only this varies per request
]
}
]
)
# First call: pays full input token cost
# Subsequent calls within the cache TTL: cached prefix billed at the discounted cache-read rate
Token reduction techniques — ordered by impact:
| Technique | Token Reduction | Implementation Effort | Risk |
|---|---|---|---|
| Prompt caching | 50–90% of static tokens | Low (add cache_point) | Minimal |
| System prompt compression | 20–40% | Medium (rewrite prompts) | Output format drift |
| Context window trimming | 30–60% | Medium (retrieval k tuning) | Reduced answer quality |
| Summarize conversation history | 40–70% of history tokens | Medium (extra FM call) | History fidelity loss |
| Model downgrade for simple queries | 50–80% cost per token | Medium (routing logic) | Capability regression |
| Output length constraints | 20–50% of output tokens | Low (max_tokens param) | Truncated responses |
Measuring token efficiency:
def log_token_efficiency(response, session_id):
usage = response['usage']
cloudwatch.put_metric_data(
Namespace='GenAI/Cost',
MetricData=[
{'MetricName': 'InputTokens', 'Value': usage['inputTokens'], 'Unit': 'Count'},
{'MetricName': 'OutputTokens', 'Value': usage['outputTokens'], 'Unit': 'Count'},
{'MetricName': 'CacheReadTokens', 'Value': usage.get('cacheReadInputTokens', 0), 'Unit': 'Count'},
{'MetricName': 'CostSavingsFromCache',
'Value': usage.get('cacheReadInputTokens', 0) * 0.000003, # $ saved
'Unit': 'None'}
],
Dimensions=[{'Name': 'SessionId', 'Value': session_id}]
)
⚠️ Exam Trap: Prompt caching is only effective when the cached prefix is identical across requests. If you include the current timestamp, a session ID, or any variable content in the system prompt, the cache never hits. Audit your system prompts for accidental variable content before expecting caching savings.
Caching economics and lifetime: cached tokens aren't free. A cache read is billed at a steep discount, about 10% of the normal input price for Claude models on Bedrock. Writing the cache can cost slightly more than a normal input token. Savings on the cached prefix are therefore roughly hit rate × ~90%. The default cache TTL is 5 minutes (some models also offer 1 hour) and resets on every hit, so a shared prefix under steady traffic stays warm and hit rates run very high. Sporadic traffic lets the cache expire between calls. Each checkpoint needs a model-specific minimum prefix length (512 to 4,096 tokens across current Claude models). Both InvokeModel and Converse support caching. If spend jumps while request counts stay flat, tokens per call have grown. Unbounded conversation history appended every turn is the usual cause; Bedrock's InputTokenCount divided by Invocations shows it.
Reflection Question: Your system prompt is 3,000 tokens and includes a daily-refreshed list of product prices (updated every morning). You process 500,000 queries per day. Calculate the daily cache-miss rate and the token waste from the daily price list update, then propose an architectural change that preserves price accuracy while maximizing cache hit rate.
