As deep neural networks stack dozens or hundreds of layers, the distribution of intermediate activations shifts constantly during training—a phenomenon known as Internal Covariate Shift. Without stabilization, activations drift into saturated regions, slowing convergence and destabilizing gradients.

Normalization layers solve this by re-centering and re-scaling intermediate outputs to maintain zero mean and unit variance. The two dominant normalization techniques in deep learning are Batch Normalization (BatchNorm) and Layer Normalization (LayerNorm).

While both apply the exact same mathematical standardization formula, they differ fundamentally in which axis of the tensor they normalize across.

1. The Core Standardization Formula

Regardless of the normalization type, every normalization layer transforms an input tensor $x$ into a normalized output $\hat{x}$, followed by a learnable affine transformation:

$$\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}$$

$$y = \gamma \hat{x} + \beta$$

Where:

  • $\mu$ and $\sigma^2$ are the mean and variance computed across the chosen normalization axis.
  • $\epsilon$ is a small constant (e.g., $10^{-5}$) for numerical stability.
  • $\gamma$ (scale) and $\beta$ (shift) are learnable parameters that allow the network to undo the normalization if the optimal representation requires a different distribution.

2. Batch Normalization (BatchNorm)

Introduced primarily for Convolutional Neural Networks (CNNs), Batch Normalization computes the mean and variance across the mini-batch dimension independently for each feature channel.

How It Works

Given an input batch of shape $(N, C, H, W)$—where $N$ is batch size, $C$ is channels, and $H, W$ are spatial dimensions—BatchNorm computes a single $\mu_c$ and $\sigma_c^2$ for each channel $c$ across all $N$ samples in the current mini-batch and all spatial positions $(H, W)$.

Batch Normalization (Normalizes across the Batch N per feature/channel)
Sample 1: [ Feature A ] [ Feature B ] [ Feature C ]
Sample 2: [ Feature A ] [ Feature B ] [ Feature C ]
Sample 3: [ Feature A ] [ Feature B ] [ Feature C ]
               |             |             |
               v             v             v
          (Mean/Var A)  (Mean/Var B)  (Mean/Var C)