2.4.4. KQL for Real-Time Processing

💡 First Principle: KQL (Kusto Query Language) is optimized for time-series analysis. It excels at aggregating and analyzing streaming data with minimal latency.

Common KQL Patterns

// Filter recent events
TemperatureReadings
| where timestamp > ago(1h)
| where temperature > 100

// Aggregate by time bucket
TemperatureReadings
| summarize AvgTemp = avg(temperature) by bin(timestamp, 5m), deviceId

// Detect anomalies
TemperatureReadings
| summarize AvgTemp = avg(temperature), StdDev = stdev(temperature) by deviceId
| where AvgTemp > 100 or StdDev > 20

Update Policies for Ingest-Time Transformations

• Concept: Transform data automatically during ingestion
• Benefit: Data lands pre-transformed, reducing query complexity
• Use Case: Parsing JSON, calculating derived fields, filtering

⚠️ Common Pitfall: Using T-SQL for KQL database transformations. KQL databases use KQL, not T-SQL. T-SQL is for the Fabric Data Warehouse.