5.1.2. Data Storage with Azure Data Lake Gen2
💡 First Principle: Data Lake Gen2 combines the scalability of Blob Storage with the file-system semantics (directories, permissions) needed for analytics workloads. The key feature is Hierarchical Namespace—without it, you're just using expensive Blob Storage.
Key Feature: Hierarchical Namespace
Without hierarchical namespace (Blob): Renaming a "folder" requires renaming every blob with that prefix (slow, expensive). With hierarchical namespace (Data Lake Gen2): Renaming a folder is a single atomic operation (fast, cheap).
Visual: Data Lake Zone Architecture
The Medallion Architecture
💡 First Principle: Raw data is untrustworthy and business-ready data is expensive to produce, so a lake stages data through quality tiers rather than cleaning it all at once.
| Layer | Also called | What lives here |
|---|---|---|
| Bronze | Raw | Exactly what the source sent, untouched, append-only — your audit trail |
| Silver | Cleansed / Conformed | Validated, de-duplicated, type-corrected, joined into consistent entities |
| Gold | Curated / Business-ready | Aggregated, modelled tables that reports and ML read directly |
Why keep Bronze at all? Because reprocessing is inevitable. If a transformation bug is found six months later, you can rebuild Silver and Gold from Bronze. If you overwrote the raw data, the history is gone for good.
Access Control: The Other Half of Hierarchical Namespace
The hierarchical namespace is not only about fast folder renames — it is what makes directory-level security possible. Gen2 supports POSIX-style ACLs: read, write and execute permissions set on individual directories and files, inherited by new children.
That gives two layers working together:
- Azure RBAC grants coarse access at the storage-account or container level — who may touch this account at all.
- POSIX ACLs grant fine-grained access inside the container — this team reads
/raw/finance, that team cannot.
Flat Blob Storage has no directories to attach permissions to, so it cannot express the second layer. When a scenario asks for folder structures with per-folder or per-file permissions, that requirement alone rules out plain Blob Storage and selects Data Lake Storage Gen2.
