In real-world machine learning, raw tabular datasets are rarely clean. Sensor logs drop readings, users skip optional form fields, extreme anomalies skew distributions, and text categories cannot be fed directly into matrix multiplication.
Garbage in guarantees garbage out. Before training any algorithm, data engineers must apply three foundational preprocessing steps: handling missing values, managing outliers, and encoding categorical variables.
1. Handling Missing Values
Before filling or dropping NaN values, identify why the data is missing—because the mechanism dictates the right fix:
- MCAR (Missing Completely at Random): Missingness has zero relationship with any variable (e.g., a random packet drop). Safe to drop or impute simply.
- MAR (Missing at Random): Missingness depends on another observed column (e.g., younger respondents skip income fields more often). Multivariate imputation works best.
- MNAR (Missing Not at Random): Missingness depends on the unobserved value itself (e.g., high-debt applicants leave credit fields blank). The missingness itself carries predictive signal—always add a binary missing-indicator flag (
is_missing = 1).
Common Remediation Strategies
| Strategy | Best Used When | Key Caveat / Risk |
|---|---|---|
| Row / Column Deletion | $<5\%$ rows missing (MCAR), or a column is $>60\%$ empty | Discards valuable data; introduces bias if not MCAR |
| Mean / Median Imputation | Numerical features (use Median if skewed) | Shrinks feature variance and distorts correlations |
| Mode Imputation | Categorical features with a clear dominant class | Over-represents the most frequent category |
| KNN / Iterative (MICE) | Features have strong inter-column correlations | Computationally slower; sensitive to feature scaling |
| Missing Indicator Flag | Data is MNAR (missingness itself is informative) | Increases dimensionality slightly |
Golden Rule: Always compute imputation statistics (mean, median, mode) strictly on the training split, then apply those saved values to validation and test sets to prevent data leakage.
2. Detecting and Managing Outliers
Outliers are extreme observations that deviate sharply from the rest of the distribution. Depending on the domain, an outlier is either corrupted noise (a sensor glitch reporting age as 999) or critical signal (a fraudulent credit card transaction).
Detection Methods
- Z-Score (Standard Score): Measures how many standard deviations a point lies from the mean: $$Z = \frac{x - \mu}{\sigma}$$ Values with $\vert{}Z\vert{} > 3$ are flagged as outliers. Best for roughly Gaussian (bell-shaped) distributions.
- IQR (Interquartile Range) Method: Uses quartiles ($Q_1 = 25\text{th percentile}$, $Q_3 = 75\text{th percentile}$), making it robust against already-skewed data: $$\text{IQR} = Q_3 - Q_1$$ $$\text{Lower Bound} = Q_1 - 1.5 \times \text{IQR}, \quad \text{Upper Bound} = Q_3 + 1.5 \times \text{IQR}$$
- Model-Based Detection: Algorithms like Isolation Forest or DBSCAN isolate multivariate anomalies across high-dimensional feature spaces.
Treatment Strategies
- Drop: Only when verified as a data entry error or physical impossibility.
- Winsorization (Capping): Clip extreme values at fixed percentiles (e.g., capping values above the 99th percentile to the 99th percentile value) to preserve sample count while taming variance.
- Log / Power Transformations: Applying $\log(1 + x)$ or Box-Cox transforms compresses right-skewed tails smoothly.
- Tree-Based Models: Decision trees, Random Forests, and XGBoost split on rank order thresholds rather than Euclidean distances, making them naturally immune to feature outliers.
3. Categorical Encoding: Converting Text to Numbers
Machine learning equations operate on numerical tensors. Categorical encoding translates discrete text labels into numbers without introducing false mathematical assumptions.
Nominal (No Order) ---> ["Red", "Blue", "Green"] ---> One-Hot Encoding ---> [1,0,0], [0,1,0], [0,0,1]
Ordinal (Ordered) ---> ["Low", "Med", "High"] ---> Ordinal Encoding ---> 0, 1, 2
High-Cardinality ---> [10,000+ Zip Codes] ---> Target / Frequency ---> Smoothed Numeric Score