3.2.3. Vector Indexes and Vectorization Concepts

💡 First Principle: Vector databases store data as high-dimensional numerical representations (embeddings) and enable similarity search — finding items that are "close" in meaning rather than exact matches. This is the foundation for AI-powered search, Retrieval Augmented Generation (RAG), and recommendation systems. The v1.1 syllabus added this topic because data engineers increasingly build pipelines that feed vector stores for generative AI applications.

A vector embedding is a list of numbers (e.g., 1,536 floating-point values) that represents the semantic meaning of text, images, or other content. Similar items have similar embeddings — a search for "cloud computing" also finds documents about "serverless infrastructure" because their embeddings are nearby in vector space.

Vector index types determine how the database organizes embeddings for fast retrieval:

HNSW (Hierarchical Navigable Small World) — builds a multi-layered graph connecting similar vectors. Provides excellent search accuracy and speed, but uses more memory. Best for workloads where search quality matters most.

IVF (Inverted File Index) — partitions vectors into clusters, then searches only the most relevant clusters. Uses less memory than HNSW but may sacrifice some accuracy. Best for large-scale workloads where memory is constrained.

AWS services for vector search:

Amazon Bedrock Knowledge Bases — managed RAG service that handles embedding generation, vector storage, and retrieval. The simplest option for building a RAG pipeline.

Amazon Aurora PostgreSQL with pgvector — adds vector similarity search to a relational database. Use when you need vector search alongside traditional SQL queries in the same database.

Amazon OpenSearch Service — supports k-nearest-neighbor (k-NN) vector search alongside its traditional text search capabilities.

⚠️ Exam Trap: The exam tests vectorization concepts, not deep implementation details. Know what embeddings are, why vector indexes exist (approximate nearest neighbor search), and which AWS services support vector search. You won't need to implement HNSW from scratch, but you should know when HNSW vs IVF is more appropriate.

Reflection Question: A company wants to build a customer support chatbot that answers questions based on internal documentation. The architecture uses Amazon Bedrock for the LLM. What role does a vector store play, and which AWS service could provide it?