2.4.1.2. Designing for NoSQL Database Workloads (Partitioning, Consistency Models)

2.4.1.2. Designing for NoSQL Database Workloads (Partitioning, Consistency Models)

💡 First Principle: Achieving massive scale and performance with "NoSQL" databases requires a deep understanding of data distribution (partitioning) and the application-level implications of different data consistency models.

Scenario: You are designing a real-time gaming leaderboard application that uses "Amazon DynamoDB". The leaderboard needs to handle millions of reads and writes per second. Updates to player scores must be immediately reflected globally for all users, but displaying the full leaderboard can tolerate a slight delay.

"NoSQL" databases offer flexibility, horizontal scalability, and often higher performance for specific access patterns compared to relational databases, making them ideal for web, mobile, gaming, and "IoT" applications.

• "Amazon DynamoDB": A key-value and document database designed for high-performance at any scale.
  • Partitioning (Key Concepts):
    • Partition Key (Hash Attribute): Determines the logical and physical partitions where data is stored. Critical for even data distribution to avoid "hot spots" and ensure performance.
    • Sort Key (Range Attribute): Defines the order of items within a partition and allows for composite primary keys.
    • Secondary Indexes ("GSI"/"LSI"): Allow flexible query patterns beyond the primary key. "Global Secondary Indexes (GSI)" can have different partition/sort keys from the table, enabling diverse queries but impacting consistency and cost.
  • Consistency Models:
    • Eventually Consistent Reads: Data from a replica might not reflect the most recent write (within a few seconds). Default for most reads. More performant, lower latency.
    • Strongly Consistent Reads: Returns the most up-to-date data, reflecting all prior writes. Higher latency, can consume more provisioned throughput.
• "Amazon DocumentDB": "MongoDB"-compatible, for document workloads.
• "Amazon Neptune": Graph database.
• "Amazon Timestream": Time series database.

Visual: DynamoDB Partitioning & Consistency Models

⚠️ Common Pitfall: Choosing a poor partition key in "DynamoDB". A key that doesn't have high cardinality (many unique values) will cause all traffic to hit a single physical partition, creating a "hot partition" that throttles requests and negates the scalability benefits of the service.

Key Trade-Offs:

• Consistency vs. Performance/Cost: "Strongly consistent reads" in "DynamoDB" provide the most up-to-date data but at the cost of higher latency and consuming double the read capacity units ("RCUs") compared to "eventually consistent reads".

DynamoDB Patterns the Exam Tests:

• Choosing consistency per use case: Use strongly consistent reads only where a just-written value must be visible (a profile edit the user is looking at) and eventually consistent reads for tolerant views (a leaderboard), saving half the read capacity. Strongly consistent reads are unavailable on GSIs and are not served from "DAX".
• Query vs. Scan: A "Query" uses the key; filtering on a non-key attribute forces a full-table "Scan" (and a filter expression still reads, and bills for, every item scanned). The fix is a Global Secondary Index keyed on that attribute so the access pattern becomes a Query. More RCUs, on-demand mode or a cache do not remove the scan.
• Capacity modes: On-demand capacity needs no capacity planning and scales to unpredictable traffic with pay-per-request pricing; provisioned mode suits steady, forecastable load. With key-based access, DynamoDB serves single-digit-millisecond reads and writes at any scale without servers, which is why it fits shopping carts, session-like data, IoT write streams and serverless JSON storage.
• DynamoDB Streams: An ordered change log (24-hour retention) of item changes that can trigger "Lambda", for example to replicate rarely changing reference data into account-local tables or to feed other systems.
• Global Tables: Multi-Region, active-active replication with local low-latency reads and writes in every replica Region (in the default multi-Region eventual consistency mode, conflicts resolve last-writer-wins). It is a table-level feature, not a "read replica", and it replicates all items to every replica Region, so its replica Regions define where the data lives. For data residency, a Region where the data may not be stored must never hold a replica of it: scope each table (or Global Table) to the Regions permitted for its data and route each user to the right one, for example with "Route 53" geolocation routing.

Reflection Question: How would you design the "DynamoDB" table's primary key (partition/sort key) and choose the appropriate consistency model for reads ("strongly consistent" vs. "eventually consistent") to meet the application's specific performance and data freshness requirements for both player score updates and leaderboard display in a real-time gaming application?