Black Friday sale: an extra 40% off every course until Monday, November 30.New Year sale: an extra 30% off every course until Friday, January 1.Memorial Day sale: an extra 25% off every course until Monday, May 31.SysAdmin Day sale: an extra 25% off every course until Sunday, August 1.Back to School sale: an extra 25% off every course until Sunday, August 22.Labor Day sale: an extra 25% off every course until Monday, September 6.Black Friday sale: an extra 40% off every course until Monday, November 29.New Year sale: an extra 30% off every course until Saturday, January 1.Choose your certification →

Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.

2.4.1.6. Lakehouse Architectures with Apache Iceberg (S3 Tables, Lake Formation, Redshift Spectrum)

2.4.1.6. Lakehouse Architectures with Apache Iceberg (S3 Tables, Lake Formation, Redshift Spectrum)

💡 First Principle: A lakehouse keeps one copy of the data in open formats on Amazon S3 and adds the table guarantees (transactions, schema evolution, time travel) that used to require loading it into a warehouse, so many engines can share the same governed tables.

  • Data lake foundations: S3 holds the data in zones (raw, curated, consumption); the "AWS Glue Data Catalog" holds table definitions; "AWS Lake Formation" grants database, table, column, row and cell-level permissions that Athena, Redshift Spectrum, EMR and Glue enforce.
  • Why a table format: plain Parquet files in Hive-style partitions are fine for appends, but a row-level update or delete (a GDPR erasure, a CDC upsert) means rewriting whole partitions, concurrent writers can conflict, and readers can see half-written data. Apache Iceberg tracks every table version in metadata (snapshots and manifests), which gives ACID transactions, row-level UPDATE/DELETE/MERGE, schema and partition evolution without rewriting data, hidden partitioning, and time travel or rollback to earlier snapshots.
  • Engines: Athena (SQL including MERGE INTO, time travel, OPTIMIZE and VACUUM), EMR and Glue (Spark), Redshift (querying Iceberg tables registered in the Data Catalog), and Amazon Data Firehose, which can deliver streams into Iceberg tables.
  • Where the Iceberg tables live:
    • Self-managed in general purpose buckets: you own table maintenance. Glue Data Catalog table optimizers automate compaction (merging small files toward a target size), snapshot retention and orphan file deletion. Do not let S3 Lifecycle expiration rules touch the table paths, because they can delete files that live snapshots still reference.
    • "Amazon S3 Tables": a table bucket stores tables in Iceberg format and S3 runs compaction, snapshot management and unreferenced file removal automatically, with higher transaction rates than self-managed tables. Access uses the separate s3tables IAM namespace (Block Public Access is always on). Integrating table buckets with the Glue Data Catalog lets Athena, Redshift and EMR discover the tables, and Lake Formation governs them.
  • Warehouse plus lake: "Amazon Redshift" keeps hot, heavily joined data in managed storage and uses Redshift Spectrum (or Data Catalog integration) to join it with S3 data in place, so the long tail never needs loading.
RequirementDesign
Row-level deletes and upserts on S3 data, several enginesIceberg tables in the Data Catalog
Same, with the least table maintenanceS3 Tables
Join warehouse tables with S3 data without loading itRedshift Spectrum or Data Catalog queries from Redshift
Column- and row-level access for analysts in many accountsLake Formation permissions (see 2.4.1.8)

⚠️ Exam Trap: Running a Glue crawler over Parquet files and expecting Athena to support transactional UPDATE/DELETE or time travel. Those need a table format such as Iceberg; a crawler only records schema and partitions.

See how it connects
Alvin Varughese
Written byAlvin Varughese
Founder•26 professional certifications earned