2.4.3. Lightweight Transforms with Lambda and Redshift SQL

💡 First Principle: Not every transformation needs a distributed processing engine. For small data volumes (under ~10 GB), simple logic (format conversion, field mapping, validation), or in-database operations, Lambda functions and Redshift SQL are lighter, faster, and cheaper alternatives to spinning up Spark.

Lambda for transformation shines in event-driven, low-volume scenarios. When a single file lands in S3 and needs light processing — parsing JSON, extracting fields, validating data quality, converting a small CSV to Parquet — Lambda handles it in seconds with no cluster startup time. Lambda is also the go-to for Kinesis Data Firehose transformations (records pass through a Lambda function before delivery).

Key Lambda constraints the exam tests: 15-minute maximum execution time (won't work for long-running transforms), 10 GB maximum memory (constrains in-memory data processing), and deployment package size limits (250 MB uncompressed, 50 MB compressed, though Lambda layers add more). Lambda functions can also mount EFS file systems for temporary storage beyond /tmp's 10 GB limit.

Redshift SQL for transformation is optimal when data is already in Redshift or is being loaded there. Redshift stored procedures and materialized views can perform complex transformations: joins, aggregations, window functions, pivots, and data type conversions — all within the warehouse. This eliminates the need to extract data, transform it externally, and reload it.

The v1.1 syllabus also includes LLM integration for data processing (Skill 1.2.10). This refers to using large language models via Amazon Bedrock to transform unstructured data — extracting entities from text, classifying documents, summarizing content, or generating structured metadata from unstructured inputs. The integration pattern typically involves Lambda invoking the Bedrock API as part of a data pipeline.

⚠️ Exam Trap: Lambda's 15-minute timeout is a hard limit. If a transformation processes files larger than a few GB or requires complex multi-step Spark logic, Lambda is not the right choice — even if the question emphasizes "serverless." In these cases, Glue (also serverless, no timeout for batch jobs) is the correct answer.

Reflection Question: A pipeline receives individual JSON records from an API. Each record is 5 KB and needs three fields extracted and written to DynamoDB. Expected volume: 100 records/minute. Should you use Lambda, Glue, or EMR? Why?