3.4.2. Eventstreams and Event Processors
💡 First Principle: Eventstreams route events from sources to destinations, and event processors provide no-code transformations during routing. Think of it like a mail sorting facility that can also open envelopes, remove junk mail, and repackage before delivery.
Scenario: IoT sensors send data with unnecessary fields (device diagnostics). The event processor removes these columns before storing in the lakehouse, reducing storage costs.
Eventstream Sources
- Azure Event Hubs
- Azure IoT Hub
- Custom applications (SDK)
- Sample data (for testing)
Eventstream Destinations
- Lakehouse (Delta tables)
- KQL Database
- Derived Eventstream (chaining)
- Activator (rules that fire actions, such as an email/Teams alert or starting a pipeline, when a condition like a temperature threshold is met)
- Custom endpoint
Event Processor Operations
| Operation | Purpose | Use Case |
|---|---|---|
| Filter | Remove unwanted events | Drop diagnostic messages |
| Manage fields | Add, remove, rename, or change the type of fields; can promote a nested property (e.g., device.id) to a top-level field | Remove unnecessary columns; flatten nested JSON |
| Aggregate | Compute statistics | Count events per minute |
| Group by | Partition events | Separate by device type |
| Union | Combine streams | Merge multiple sources; only fields that match in name and data type are kept, others are dropped |
| Join | Combine two streams on a matching condition, or a stream with reference data (a lakehouse Delta table) | Enrich transactions with merchant or customer details in flight |
| Expand | Create one row per element of an array field | Explode an array of readings into rows (not for nested objects) |
Derived streams and routing: a derived stream is a named output of an operator path (for example, Filter → derived stream). One eventstream can branch into several paths, each with its own filter, aggregation, and destination: high-priority events filtered to an Eventhouse, windowed summaries to a KQL table, raw events passed through to a lakehouse, threshold breaches sent to Activator. The source is read only once; separate eventstreams on the same source duplicate ingestion. Derived streams also appear in the Real-Time hub for others to use.
Source and destination pitfalls:
- Consumer lag vs. retention: Event Hubs keep events only for the hub's retention period (1 day on Basic, up to 7 on Standard, up to 90 on Premium and Dedicated). If consumer-group lag keeps growing and the oldest unread events outlive retention, they are deleted before the Eventstream reads them, leaving a silent, unrecoverable gap. Raise the eventstream's throughput level (4.3.5) or the hub's retention before that happens.
- Duplicates in a KQL destination: a KQL database ingests every record it is sent and does not deduplicate on ingest. If a change (such as a new schema field) leaves the same events reaching a table by two routes, such as a second destination or ingestion mapping alongside the original, every event lands twice. Check routes and mappings after schema changes.
⚠️ Exam Trap: Azure Data Factory is designed for batch orchestration, not real-time event processing. For streaming workloads, use Eventstreams. Questions mentioning "real-time" or "continuous" processing require streaming components, not ADF.
