5.4.5. Token, Embedding and Vector Storage Cost Patterns
💡 First Principle: A retrieval-augmented system has three meters running: tokens for the generator, tokens for the embedding model, and storage plus compute for the vector store. Each meter has its own driver, so each needs its own metric. A bill that rises with no traffic increase usually means one of the three is doing work it did not need to.
Token usage patterns. Break token usage down by model and by application before acting (5.4.4 shows how). Patterns worth recognising:
| Pattern | Likely cause | Response |
|---|---|---|
| Input tokens per request climb week over week | Conversation history or retrieved context growing unbounded | Summarise history, cap retrieved chunks, re-rank and keep the top results |
| High input tokens, identical prefix on every call | Static instructions resent in full | Prompt caching; check CacheReadInputTokenCount rises after enabling |
| Output tokens far above what users read | Verbose prompt or generous maxTokens | Constrain format and length |
| Tokens per session far above peers | Agent loop re-planning or retrying tools | Iteration limits, fix the failing tool (5.4.3) |
Embedding computation cost. Embedding models are billed per input token, so cost scales with how much text you embed and how often you re-embed it:
- Sync incrementally. Amazon Bedrock Knowledge Bases syncs are incremental: only documents added, modified or deleted since the last sync are processed. A pipeline that deletes and rebuilds the whole index each night pays to re-embed an unchanged corpus.
- Batch large backfills. Batch inference supports embedding models such as Amazon Titan Text Embeddings V2, at the batch discount, for initial loads or re-indexing after a model change.
- Chunk deliberately. Heavy chunk overlap multiplies the tokens embedded for the same source text. Tune overlap against retrieval quality rather than maximising it.
- Embed once, reuse. Cache the embedding of repeated queries or documents instead of recomputing it.
Vector storage optimisation. Vector storage cost scales with number of vectors × dimensions × bytes per dimension, plus the compute that serves queries:
- Fewer dimensions. Titan Text Embeddings V2 can output 1,024 (default), 512 or 256 dimensions. Halving dimensions halves raw vector storage; validate retrieval quality before committing.
- Smaller encoding. Titan Text Embeddings V2 can also return binary embeddings (
embeddingTypes), which a vector store that supports binary vectors can keep at one bit per dimension instead of a 32-bit float. - Cheaper storage tier. Amazon S3 Vectors provides purpose-built vector storage with sub-second queries for infrequent access and as low as about 100 ms for frequent queries; AWS states it can reduce the cost of uploading, storing and querying vectors by up to 90%. It suits large corpora queried occasionally. For high queries-per-second, low-latency search, OpenSearch remains the fit, and an S3 Vectors index can be exported to OpenSearch Serverless for that purpose.
- Bound the compute. OpenSearch Serverless scales OpenSearch Compute Units (OCUs) within the minimum and maximum you set per collection group; the maximum is your spending cap. Monitor
SearchOCUandIndexingOCU. On provisioned OpenSearch domains,KNNGraphMemoryUsageshows how close vector graphs are to the memory the k-NN plugin can use.
⚠️ Exam Trap: "Use a larger embedding dimension to improve accuracy" is a cost answer in disguise. More dimensions mean more storage and more memory for every vector, forever. When the scenario's constraint is cost, reduce dimensions or switch to binary embeddings, and verify retrieval quality with an evaluation.
⚠️ Exam Trap: When embedding cost spikes after a scheduled job but document volume has not changed, the cause is usually full re-ingestion rather than incremental sync. Check what the pipeline re-embeds before changing models.
