2.5. Data Formats and Conversion

💡 First Principle: Choosing the right data format is one of the highest-leverage decisions in data engineering — it affects query speed, storage cost, and processing time simultaneously. Think of it like choosing between storing books in a library by page order (row-based) or by chapter topic (columnar): how you organize the data determines how efficiently you can find what you need.

Consider a data lake that stores everything as CSV: analysts scanning 500 GB tables wait 10 minutes per query. For instance, converting to Parquet with snappy compression cuts that to 30 seconds — a 20x improvement from a format change alone.

The wrong format choice silently multiplies costs. What happens when a team queries CSV files in Athena? They pay to scan every byte of every column in every row — even if they only need three columns out of fifty. Without format conversion, a data lake that costs $500/month to query could cost $50/month with Parquet — same data, same queries, 90% less money. The format decision happens once at ingestion time, but its cost impact compounds with every query thereafter.

The exam tests format selection frequently, often disguised as cost optimization or performance tuning questions. When a question mentions "reduce Athena query costs" or "improve query performance on S3 data," format conversion is almost always part of the answer. Can you justify why Parquet outperforms CSV for analytics but CSV is still the right landing format for raw data?