4.3.5. Eventstream and Eventhouse Optimization
💡 First Principle: Real-time system optimization focuses on latency reduction and throughput improvement while maintaining data integrity. The trade-offs differ from batch: you're optimizing for time, not just resource efficiency. A 5-second delay in batch analytics is invisible; in fraud detection, it's the difference between blocking a transaction and explaining it to the customer afterward.
Eventstream Optimization
| Lever | Effect | Trade-Off |
|---|---|---|
| Batch size | Larger batches improve throughput | Increases end-to-end latency |
| Parallelism (partitions) | More partitions enable parallel processing | More resources consumed |
| Filter early | Remove unwanted events before routing | Reduces downstream load |
| Schema projection | Remove unused columns at ingestion | Less data stored and processed |
| Derived streams | Route different event types separately | Targeted optimization per stream |
| Event throughput level | Low / Medium / High setting scales the eventstream's sources and destinations; too low a level for the incoming rate builds a backlog | Higher level consumes more capacity (CU) |
| Transformation complexity | Fewer chained operators in the event processor mean less per-event processing time | Heavier logic moves to the destination (e.g., a KQL update policy) |
Optimization Pattern:
Source → [Filter unwanted] → [Project needed columns] → [Route to destinations]
Filtering and projecting at the eventstream level is always cheaper than filtering at the destination.
Eventhouse (KQL Database) Optimization
| Policy | Purpose | Configuration | Impact |
|---|---|---|---|
| Retention policy | Auto-delete data after a period | .alter table T policy retention '{"SoftDeletePeriod": "365.00:00:00"}' | Reduces storage, keeps KQL fast |
| Caching policy | Keep hot data in memory/SSD | .alter table T policy caching hot = 30d | Fast queries on recent data |
| Partitioning policy | Organize data by column | .alter table T policy partitioning '{"PartitionKeys": [...]}' | Improves filter performance |
| Ingestion batching | Control how events are batched | .alter table T policy ingestionbatching '{"MaximumBatchingTimeSpan": "00:00:30"}' | Balance latency vs. efficiency |
| Streaming ingestion | Low-latency writes for small, continuous volumes | .alter database DB policy streamingingestion enable | Events become queryable in seconds instead of waiting for a batch to seal; lower throughput than batched (queued) ingestion |
| Materialized view | Aggregation that is pre-computed and maintained incrementally in the background | .create materialized-view LatestByDevice on table T { T | summarize arg_max(Timestamp, *) by DeviceId } | Repeated aggregations and latest-value lookups return fast and are always up to date |
Caching Tiers:
Hot cache (SSD/Memory) ← Fastest queries, most expensive
└── Frequently queried recent data (e.g., last 30 days)
Cold storage ← Slower queries, cheapest
└── Historical data (e.g., older than 30 days)
Key Decision: Retention vs. Caching
- Retention controls how long data exists (deleted after period)
- Caching controls how fast data is queried (hot vs. cold)
- Data can be retained for 1 year but only cached (hot) for 30 days
Streaming vs. batched ingestion: When ingestion is falling behind (a throughput problem), send larger batches, because queued ingestion is optimized for bulk. When events must be queryable within seconds (a latency problem), enable streaming ingestion.
KQL Query Best Practices:
- Filter on the datetime column first (
where Timestamp > ago(1h)) so the engine skips data outside the time range. Filter on the column the data is actually time-ordered by (usually ingestion or event time). projectonly the columns you need, after the filters.takebeforewherereturns arbitrary rows, and KQL has noselect.- To see execution statistics, use
.show queries(recent queries with duration, CPU and memory). KQL doesn't use SQL-styleEXPLAIN.
⚠️ Exam Trap: Setting the caching policy longer than the retention policy is wasteful—data is deleted by retention before it ages out of cache. Always ensure caching period ≤ retention period.
⚠️ Common Pitfall: Not setting retention policies on high-volume streaming tables. Without retention, storage grows indefinitely and query performance degrades as KQL scans more data.
