2.2.2. Feature Engineering: Scaling, Binning, and Log Transforms
💡 First Principle: Feature engineering reshapes raw data into representations that make patterns easier for the model to learn. A good feature doesn't add new information—it restructures existing information so the model can exploit it efficiently. The exam tests whether you know which transformation to apply for which data characteristic.
Scaling and Normalization:
Why does scaling matter? Algorithms that compute distances (KNN, K-Means, SVM) or use gradient-based optimization (neural networks, linear regression) are sensitive to feature magnitudes. A feature ranging 0–1,000,000 dominates one ranging 0–1, not because it's more important, but because it's numerically larger.
| Technique | Formula | When to Use | Preserves Outliers? |
|---|---|---|---|
| Standard scaling (Z-score) | (x − μ) / σ | Gaussian-distributed features, gradient-based algorithms | Yes (outliers affect μ, σ) |
| Min-max normalization | (x − min) / (max − min) | Need bounded [0,1] range, neural networks | No (outliers compress range) |
| Robust scaling | (x − median) / IQR | Features with significant outliers | Yes (IQR resistant to outliers) |
| Max-abs scaling | x / | max |
Log Transformation: For features with right-skewed distributions (income, transaction amounts, web page views), log transformation compresses the tail and makes the distribution more Gaussian-like. This helps linear models and can improve tree-based model performance by reducing the influence of extreme values.
Binning (Discretization): Converting continuous features into categorical bins. "Age" becomes "18-25," "26-35," etc. This is useful when the relationship between the feature and target is non-linear and step-wise, or when you want to reduce noise in a continuous variable.
Feature Splitting: Extracting multiple features from one. A "datetime" column becomes "hour," "day_of_week," "month," "is_weekend," "is_holiday." An "address" column becomes "city," "state," "zip_code." This decomposes complex features into components the model can learn from individually.
⚠️ Exam Trap: Tree-based algorithms (XGBoost, Random Forest) are largely invariant to feature scaling—they split on value thresholds, so the magnitude doesn't matter. If a question describes using XGBoost and asks about scaling, it may be a distractor. Scaling matters most for distance-based and gradient-based algorithms.
Reflection Question: A dataset contains a "purchase_amount" feature with values ranging from $0.50 to $50,000, with 90% of values below $100. Which transformation would be most appropriate, and why would standard scaling alone be problematic?
