Black Friday sale: an extra 40% off every course until Monday, November 30.New Year sale: an extra 30% off every course until Friday, January 1.Memorial Day sale: an extra 25% off every course until Monday, May 31.SysAdmin Day sale: an extra 25% off every course until Sunday, August 1.Back to School sale: an extra 25% off every course until Sunday, August 22.Labor Day sale: an extra 25% off every course until Monday, September 6.Black Friday sale: an extra 40% off every course until Monday, November 29.New Year sale: an extra 30% off every course until Saturday, January 1.Choose your certification →

Copyright (c) 2026 MindMesh Academy. All rights reserved. This content is proprietary and may not be reproduced or distributed without permission.

2.4.1.7. Serverless, Streaming and Batch Analytics Pipelines (Kinesis, Lambda, Glue, MSK, EMR, AppFlow)

2.4.1.7. Serverless, Streaming and Batch Analytics Pipelines (Kinesis, Lambda, Glue, MSK, EMR, AppFlow)

💡 First Principle: Pick the processing model by the latency the consumer needs (seconds, minutes or hours) and the operational load the team can carry; serverless building blocks remove capacity planning, and managed clusters give control when the transformation needs it.

Serverless pipeline building blocks:
  • Ingest: Kinesis Data Streams (ordered shards, replay), Amazon Data Firehose (managed delivery), S3 event notifications routed through EventBridge.
  • Transform in Firehose: buffering by size or time writes fewer, larger objects; a Lambda function can transform records; record format conversion turns JSON into Parquet or ORC using a schema in the Glue Data Catalog; dynamic partitioning builds S3 prefixes from keys inside each record (for example model= and date=), so Athena scans less data.
  • Lambda on streams: an event source mapping reads each shard in order, in batches (with an optional batching window), and a parallelization factor (up to 10) processes more batches per shard while keeping records with the same partition key in order. Tumbling windows (up to 15 minutes) carry state between invocations for simple aggregations. Because processing is in order, a record that keeps failing blocks its shard until it expires; configure bisect batch on function error, maximum retry attempts, maximum record age, partial batch responses (ReportBatchItemFailures) and an on-failure destination so bad records are set aside and the shard keeps moving.
  • AWS Glue: crawlers populate the Data Catalog, Spark and Python shell jobs transform data, Glue streaming jobs process streams, and Glue workflows or Step Functions orchestrate multi-step jobs. Athena queries the results in place. Amazon Managed Service for Apache Flink handles stateful stream processing (windows, joins, pattern detection).
Real-time and batch analytics services:
  • "Amazon MSK": managed Apache Kafka (provisioned or Serverless) for teams with Kafka producers, consumers and connectors (MSK Connect). Choose it over Kinesis when the Kafka API and ecosystem must be kept; Kinesis is the simpler AWS-native stream.
  • Redshift streaming ingestion: a materialized view over a Kinesis data stream or an MSK/Kafka topic loads records into Redshift within seconds, with no staging in S3 and no Firehose or Glue job in between. Zero-ETL integrations replicate Aurora, RDS and DynamoDB data into Redshift for near-real-time analytics without building pipelines.
  • "Amazon EMR": Spark, Hive and Trino for large batch transformations, on EC2 clusters (full control, Spot task nodes), EMR Serverless (no cluster sizing, pay per job) or EMR on EKS.
  • "Amazon AppFlow": fully managed transfer from SaaS applications such as Salesforce, SAP, ServiceNow and Zendesk into S3 or Redshift (and back), on a schedule, on an event or on demand, with filtering, field mapping and validation and no code.

A typical design runs a hot path (Kinesis or MSK into Redshift streaming ingestion or OpenSearch for dashboards in seconds) beside a cold path (Firehose or Glue into S3 Iceberg tables, with EMR or Athena batch jobs overnight).

⚠️ Exam Trap: Raising the batch size or parallelization factor to "unstick" a Kinesis-triggered Lambda function that fails on one bad record. The record keeps blocking its shard until it is bisected out and sent to an on-failure destination or ages past the maximum record age.

See how it connects
Alvin Varughese
Written byAlvin Varughese
Founder•26 professional certifications earned