2.5. Reflection Checkpoint: Data Engineering for Machine Learning

Key Takeaways

• Data ingestion splits along two axes: batch vs. streaming (historical bulk loads vs. continuous low-latency capture) and full vs. incremental — AWS DMS and DynamoDB Streams handle change-data-capture from operational databases without impacting live OLTP performance, while Kinesis Data Streams/Firehose and MSK handle real-time streams.
• Storage choice follows access pattern, not just data type: S3 for a data lake (any format, any scale, cheapest), Redshift for a data warehouse (columnar, MPP, complex SQL aggregations), and DynamoDB/DocumentDB for NoSQL (flexible schema, single-digit-millisecond feature serving) — the same customer data might legitimately live in all three depending on how it's consumed.
• Batch ETL (Glue for serverless jobs, EMR for custom Spark/Hadoop control, Athena for ad-hoc SQL) transforms data on a schedule; streaming processing (Kinesis Data Analytics, Spark Streaming on EMR) transforms it continuously — SageMaker Data Wrangler and Processing Jobs sit downstream of both, doing the ML-specific prep general ETL doesn't.
• The Glue Data Catalog is metadata only (schema, location) — it doesn't hold the data itself; Lake Formation builds on top of it to add fine-grained (table/column/row-level) access control that would otherwise require juggling separate S3 bucket policies and IAM policies.
• IAM policies (attached to identities) and resource policies (attached to the S3 bucket or KMS key itself) both grant or deny access, and when both apply, an explicit deny always wins — this is exactly the mechanism that restricts a sensitive training-data bucket to only the SageMaker role that needs it.

Connecting Forward

Phase 2 built the data foundation: how data gets in, where it lives, how it's transformed, and who's allowed to touch it. Phase 3 picks up exactly where clean, governed data leaves off — exploring it, engineering features from it, and handling the imbalances and outliers that would otherwise sabotage model training.

Self-Check Questions

• Your team needs to capture every insert/update/delete on a live production DynamoDB table in near-real-time to keep an online feature store current, without adding load to the table itself. Which AWS mechanism does this, and why wouldn't a nightly batch export be good enough?
• A data scientist can query a table in Athena, but is unexpectedly denied access to three specific PII columns within it. Which service enforces that column-level restriction?