5.1.3. Data Processing with Azure Synapse Analytics
💡 First Principle: Azure Synapse is a unified analytics platform that combines data warehousing, big data processing, and data integration into a single service. Instead of managing separate clusters for SQL and Spark, Synapse provides both under one roof with shared governance and security.
Key Components
- Synapse SQL:
- Dedicated SQL Pool: Provisioned data warehouse (MPP). Pay per DWU.
- Serverless SQL Pool: Query data in Data Lake using T-SQL without loading it. Pay per TB scanned.
- Apache Spark Pool: Managed Spark clusters for big data processing (Python, Scala, R).
- Synapse Pipelines: Data Factory integrated within Synapse.
Visual: Azure Synapse Components
The Three Pools
| Pool | Billing model | Use it for |
|---|---|---|
| Dedicated SQL pool | Provisioned capacity, measured in Data Warehouse Units (DWU) | A always-on enterprise data warehouse with predictable, consistent performance |
| Serverless SQL pool | Pay per terabyte of data scanned; nothing to provision | Ad-hoc T-SQL exploration of files already sitting in the data lake |
| Apache Spark pool | On-demand cluster billed per vCore-hour; autoscales and auto-pauses when idle | Python, Scala or .NET processing and data engineering at scale |
Reasoning Tool: "predictable cost and consistent performance" → dedicated. "Query the lake without loading it first" → serverless. "Python and Spark" → Spark pool.
External Tables and PolyBase
PolyBase is the technology that lets a Synapse SQL pool query data held outside the database — files in Data Lake Storage or Blob Storage — using ordinary T-SQL.
An external table is the metadata that makes this work: a table definition that points at those files and describes their schema, without importing a single row. Query it and Synapse reads the files in place. It is a permanent definition, not a temporary or session-scoped object, and it stores no data of its own.
Azure Synapse Link
Synapse Link gives you near-real-time analytics over operational data with no ETL pipeline at all. Enabling it on a Cosmos DB container creates an analytical store — a column-oriented copy kept automatically in sync alongside the transactional store.
The point is isolation: analytical queries read the analytical store, so they consume no Request Units from the operational workload and cannot slow the application down. The trade-off is that the data is now held twice, so storage goes up — what you save is the pipeline, not the space.
