3.1.3. Design for Data Streaming
💡 First Principle: Ingesting and processing data in real-time as it is generated is essential for enabling immediate insights, rapid operational responses, and building event-driven architectures.
Scenario: You are designing a system to monitor smart city infrastructure. Millions of sensors continuously send traffic and environmental data. This data needs to be ingested in real-time, analyzed for anomalies, and visualized on a live dashboard.
Data streaming solves the challenge of acting on information as it is generated, rather than waiting for periodic batch processing. Its foundational purpose is to enable real-time insights and immediate operational responses.
Azure’s core data streaming services:
- Azure Event Hubs: High-throughput event ingestion, capable of collecting millions of events per second. Ideal as a "front door" for streaming data into Azure.
- Azure Stream Analytics: Real-time stream processing engine, supporting windowed aggregations, anomaly detection, and complex event processing.
- Azure IoT Hub: Specialized for IoT, enabling secure, bi-directional communication with millions of devices, handling both telemetry ingestion and device management.
Key design considerations:
- Ingestion patterns: Use partitioning to distribute load and consumer groups for parallel, independent processing of event streams.
- Processing logic: Apply windowing (time/count-based), aggregations, and real-time analytics to extract actionable insights.
- Output destinations: Route processed data to databases (e.g., Azure SQL, Cosmos DB), dashboards (Power BI), or storage (Blob Storage).
- Scalability & reliability: Architect for elastic scaling to handle variable data rates, and ensure reliability with checkpointing and geo-replication.
Design decisions in practice — the broker is chosen by what the message IS:
| Service | Use when | Consumption model |
|---|---|---|
| Event Hubs | High-volume telemetry and event streams, millions of events per second | Partitioned log; consumers read at their own offset and can replay |
| IoT Hub | Device telemetry that also needs per-device identity, bidirectional commands and device management | Built on Event Hubs semantics, plus a device registry |
| Service Bus | Discrete business messages that must each be handled exactly once | Per-message delivery with locking, dead-lettering, sessions and transactions |
| Event Grid | Reactive notifications — "something happened, go do a thing" | Push-based routing to handlers with retry |
The distinction that decides most questions is stream versus message. A stream is read by offset and can be replayed by a new consumer from the beginning; a message is delivered, locked, completed and gone. Anything requiring per-message acknowledgement, dead-lettering or ordering guarantees within a session is Service Bus territory, and anything measured in events-per-second with multiple independent readers is Event Hubs.
Capture is the archiving shortcut worth knowing: Event Hubs Capture writes the stream to Blob Storage or ADLS Gen2 in Avro automatically, removing the need for a consumer whose only job is archiving. Note it is an additional per-throughput-unit charge on the Standard tier and included on Premium and Dedicated.
⚠️ Exam Trap: Event Hubs retention is a short window measured in days, not an archive. A requirement to keep raw events for years means Capture writing to storage, not a longer retention setting.
⚠️ Common Pitfall: Using a standard database for high-throughput event ingestion. Traditional databases are not designed to handle the massive write volumes of real-time streaming data and will quickly become a bottleneck.
Key Trade-Offs:
- Real-time vs. Batch Processing: Real-time streaming provides immediate insights but can be more complex and costly to operate than traditional batch processing, which has higher latency.
Reflection Question: How does designing for data streaming (leveraging Azure Event Hubs for ingestion and Azure Stream Analytics for real-time processing) fundamentally enable real-time insights and immediate operational responses, allowing you to act on information as it is generated?
