3.4.5. Windowing Functions

💡 First Principle: Windowing functions divide continuous streams into finite chunks for aggregation—because you can't summarize infinity. Different window types serve different analytical needs: counting events per hour, calculating rolling averages, or grouping activity sessions.

Scenario: You need to: (1) Count events every 5 minutes (tumbling), (2) Calculate rolling average over overlapping intervals (hopping), (3) Group events until no activity for 30 seconds (session).

Window Types

In Spark code: window(col("eventTime"), "5 minutes") is a tumbling window; adding a slide duration, window(col("eventTime"), "5 minutes", "1 minute"), makes it overlapping (Spark calls this a sliding window; Eventstream and Stream Analytics call it hopping). session_window(col("eventTime"), "30 seconds") gives session windows. The window goes inside groupBy(...), not as a DataFrame method.

Late data and watermarks: Window on event time (the timestamp in the record), not the time Spark receives it, or late events land in the wrong window. df.withWatermark("eventTime", "10 minutes").groupBy(window("eventTime", "5 minutes")).count() accepts events up to 10 minutes late: once the watermark passes a window's end, the window is finalized, its state is released, and later events for it may be dropped. Without a watermark Spark can never tell when a window is complete, so window state grows without bound and the job eventually runs out of memory. Set the delay to the known maximum lateness; a smaller delay drops legitimately late events.

⚠️ Exam Trap: A streaming watermark is not the batch watermark from 3.1.1. The batch one is a stored high-water value (for example, max LastModifiedDate) used to select new rows on the next incremental run; the streaming one is a moving event-time threshold that bounds lateness and state.

Visual: Window Types