2.1.1. OneLake: The Unified Data Lake

💡 First Principle: OneLake works like a company-wide shared drive where every department's files automatically appear in the same place—no copying, no syncing, no "which version is correct?" debates. All Fabric workloads store data here by default, which means a lakehouse, a data warehouse, and a KQL database can all access the same underlying data without moving it.

Scenario: A data engineer creates a lakehouse, a data warehouse, and a KQL database. In traditional architectures, each would have isolated storage, requiring complex ETL to share data. In Fabric, all three store data in OneLake, enabling cross-workload analytics with a single security model.

Key Characteristics of OneLake

• Single Namespace: One logical data lake for the entire organization—no more data silos
• Delta Lake Foundation: All tabular data stored in Delta format (Parquet + transaction log) providing ACID transactions
  • Time travel: query an earlier version with VERSION AS OF / TIMESTAMP AS OF until VACUUM deletes the old files (default retention 7 days); the schema can evolve (for example with mergeSchema)
  • Formats compared: Parquet is compressed columnar binary with no transaction log; Delta is Parquet data files plus a _delta_log of JSON commits (ACID, time travel); CSV is row-oriented, delimited plain text that is human-readable and opens in spreadsheet tools, with no compression or schema; JSON / JSON Lines is row-oriented text that can express nested structures
• Automatic Data Storage: Every Fabric item stores data directly in OneLake without explicit configuration
• Hierarchical Organization: Tenant (one OneLake) → Workspace → Item (lakehouse, warehouse…) → Tables / Files folders. A capacity is the compute a workspace is assigned to, not a level in the OneLake path
• Built-in Governance: Security, lineage, and compliance inherited automatically

Lakehouse Tables vs. Files:

• Tables holds Delta tables registered in the metastore—these are what the SQL analytics endpoint and semantic models can query
• Files holds unmanaged files of any format (CSV, JSON, images)—a natural raw landing zone, not visible to the SQL analytics endpoint until loaded into a table
• Managed vs. external tables: a managed table (the default, e.g. written with saveAsTable) keeps its data files under the lakehouse Tables folder and its data and metadata both belong to the lakehouse, so DROP TABLE deletes the data files too; an external table registers metadata over files at a path you specify (a Files path or a shortcut), so dropping it leaves the files in place

Visual: OneLake Architecture

Shortcuts: Virtual Data Access

• Concept: Pointers to external data sources that appear as native OneLake data
• Types:
  • Internal Shortcuts: Point to other locations within OneLake
  • External Shortcuts: Point to Azure Data Lake Gen2, AWS S3, or Google Cloud Storage
• Benefit: Access external data without copying, maintaining single source of truth
• Always live: a query through a shortcut reads the target's current state—no snapshot, no sync lag. Optional shortcut caching (see 3.2.3) still serves the remote file whenever the source has a newer version
• Delete semantics: deleting a shortcut removes only the pointer; deleting a file or folder inside a shortcut deletes it at the target (given write permission)
• Migrations: only consumers that read through the shortcut path benefit—semantic models, dataflows or notebooks still holding the old ADLS Gen2 URL keep reading the old location until repointed
• Tables vs. Files: a shortcut created under Tables that points to Delta data shows up as a table (queryable from the SQL analytics endpoint); one under Files shows up as a folder

⚠️ Exam Trap: Shortcuts are metadata pointers—data stays in its original location. This means external data governance still applies to the source system. If someone asks about copying data, shortcuts are NOT the answer.

Key Trade-Offs:

• Centralization vs. Flexibility: OneLake centralizes governance but requires all workloads to conform to its security model
• Shortcuts vs. Copies: Shortcuts avoid duplication but depend on source system availability—if AWS S3 goes down, your shortcut is inaccessible

Reflection Question: If your organization requires data to physically reside within a specific geographic region for compliance, how would OneLake shortcuts to an external S3 bucket in a different region affect your compliance posture?