2.4.1.6. Lakehouse Architectures with Apache Iceberg (S3 Tables, Lake Formation, Redshift Spectrum)
2.4.1.6. Lakehouse Architectures with Apache Iceberg (S3 Tables, Lake Formation, Redshift Spectrum)
💡 First Principle: A lakehouse keeps one copy of the data in open formats on Amazon S3 and adds the table guarantees (transactions, schema evolution, time travel) that used to require loading it into a warehouse, so many engines can share the same governed tables.
- Data lake foundations: S3 holds the data in zones (raw, curated, consumption); the
"AWS Glue Data Catalog"holds table definitions;"AWS Lake Formation"grants database, table, column, row and cell-level permissions that Athena, Redshift Spectrum, EMR and Glue enforce. - Why a table format: plain Parquet files in Hive-style partitions are fine for appends, but a row-level update or delete (a GDPR erasure, a CDC upsert) means rewriting whole partitions, concurrent writers can conflict, and readers can see half-written data. Apache Iceberg tracks every table version in metadata (snapshots and manifests), which gives ACID transactions, row-level
UPDATE/DELETE/MERGE, schema and partition evolution without rewriting data, hidden partitioning, and time travel or rollback to earlier snapshots. - Engines: Athena (SQL including
MERGE INTO, time travel,OPTIMIZEandVACUUM), EMR and Glue (Spark), Redshift (querying Iceberg tables registered in the Data Catalog), and Amazon Data Firehose, which can deliver streams into Iceberg tables. - Where the Iceberg tables live:
- Self-managed in general purpose buckets: you own table maintenance. Glue Data Catalog table optimizers automate compaction (merging small files toward a target size), snapshot retention and orphan file deletion. Do not let S3 Lifecycle expiration rules touch the table paths, because they can delete files that live snapshots still reference.
"Amazon S3 Tables": a table bucket stores tables in Iceberg format and S3 runs compaction, snapshot management and unreferenced file removal automatically, with higher transaction rates than self-managed tables. Access uses the separates3tablesIAM namespace (Block Public Access is always on). Integrating table buckets with the Glue Data Catalog lets Athena, Redshift and EMR discover the tables, and Lake Formation governs them.
- Warehouse plus lake:
"Amazon Redshift"keeps hot, heavily joined data in managed storage and uses Redshift Spectrum (or Data Catalog integration) to join it with S3 data in place, so the long tail never needs loading.
| Requirement | Design |
|---|---|
| Row-level deletes and upserts on S3 data, several engines | Iceberg tables in the Data Catalog |
| Same, with the least table maintenance | S3 Tables |
| Join warehouse tables with S3 data without loading it | Redshift Spectrum or Data Catalog queries from Redshift |
| Column- and row-level access for analysts in many accounts | Lake Formation permissions (see 2.4.1.8) |
⚠️ Exam Trap: Running a Glue crawler over Parquet files and expecting Athena to support transactional UPDATE/DELETE or time travel. Those need a table format such as Iceberg; a crawler only records schema and partitions.