2.2.2. Batch vs. Streaming Processing

💡 First Principle: The difference between batch and streaming processing comes down to one question: "How quickly must I act on this data?" Batch processing is like mail delivery—efficient because items are grouped, but you wait for the delivery schedule. Streaming is like a phone call—immediate but requires dedicated resources listening continuously. Batch optimizes for throughput (processing large volumes efficiently). Stream optimizes for latency (processing individual events immediately).

What breaks when you choose wrong? Use batch processing for fraud detection, and by the time your nightly job runs, the fraudulent transaction is hours old and the money is gone. Use streaming for monthly payroll, and you'll pay for 24/7 infrastructure to do work that needs to happen once a month.

Scenario: A bank processes payroll for 100,000 employees. This happens once per month at 2 AM—latency is irrelevant; accuracy and completeness matter. The same bank also monitors credit card transactions for fraud—each transaction must be analyzed within milliseconds before approval.

Batch Processing

• Concept: Data is collected over time and processed in scheduled chunks
• Latency: High (minutes to hours acceptable)
• Throughput: Optimized for maximum efficiency
• Use Cases:
  • Daily/weekly/monthly reports
  • ETL jobs moving data to warehouses
  • Historical data analysis
  • Payroll processing
• Azure Services: Azure Data Factory, Azure Synapse Pipelines, Azure Databricks (batch mode)

Stream Processing

• Concept: Data is processed record-by-record as it arrives
• Latency: Low (milliseconds to seconds required)
• Throughput: May sacrifice efficiency for speed
• Use Cases:
  • Fraud detection (real-time authorization)
  • IoT sensor monitoring
  • Live dashboards
  • Social media sentiment analysis
• Azure Services: Azure Stream Analytics, Azure Event Hubs, Azure Databricks (streaming mode)

💡 Key Insight: Roles in Real-Time Architecture

• Azure Event Hubs acts as the "Ingestor" (collects and holds the streams of data).
• Azure Stream Analytics acts as the "Processor" (queries and analyzes the data in real-time as it arrives).

Visual: Batch vs. Stream Processing

Comparative Table: Batch vs. Stream

⚠️ Exam Trap: Using batch processing for fraud detection is always wrong. By the time the batch runs, the fraudulent transaction is hours old and the money is gone. Real-time decisions require stream processing.

Key Trade-Offs:

• Latency vs. Throughput: Stream processing delivers low latency but processes data less efficiently. Batch processing is highly efficient but introduces delay.
• Simplicity vs. Responsiveness: Batch jobs are simpler to build and debug. Stream processing requires handling out-of-order events, late arrivals, and state management.
• Cost vs. Immediacy: Streaming infrastructure runs continuously (higher cost). Batch infrastructure runs periodically (lower cost but delayed insights).

Reflection Question: A ride-sharing company needs to match drivers with passengers in real-time, but also wants to analyze historical ride patterns monthly. Which processing type would you use for each requirement, and why?