Before the modern deep learning revolution, researchers struggled to train neural networks deeper than three or four layers. Adding more layers—which theoretically should increase expressive power—frequently caused model training to stall completely or crash with NaN errors.
The root cause of this training instability lies in two closely related mathematical phenomena during backpropagation: Vanishing Gradients and Exploding Gradients.
Understanding why gradients decay or blow up—and knowing the architectural solutions designed to stabilize them—is essential for training deep neural networks and Recurrent Neural Networks (RNNs).
1. The Mathematical Root: Backpropagation and the Chain Rule
Neural networks learn by computing the gradient of the loss function $L$ with respect to each layer's weights $W_l$, and stepping in the opposite direction.
To propagate error signals backward from the output layer ($L$) to early hidden layers ($1$), backpropagation repeatedly applies the Calculus Chain Rule:
$$\frac{\partial L}{\partial W_1} = \frac{\partial L}{\partial a_n} \cdot \frac{\partial a_n}{\partial a_{n-1}} \cdot \frac{\partial a_{n-1}}{\partial a_{n-2}} \cdots \frac{\partial a_2}{\partial a_1} \cdot \frac{\partial a_1}{\partial W_1}$$
Because each layer transformation involves multiplying by the weight matrix $W_l$ and the derivative of the activation function $f'(z_l)$, computing the gradient at layer 1 involves multiplying $n$ Jacobian terms together.
Output Layer (Layer N) ---> Layer N-1 ---> ... ---> Layer 1 (Input)
[Gradient] x (W * f') x (W * f') [Final Update]
