Standard Stochastic Gradient Descent (SGD) applies a fixed, uniform step size across every parameter. In practice, real-world loss landscapes are rarely neat, symmetric bowls. Instead, they are filled with ravines—areas where the loss curves steeply in one dimension but gently in another—and vast flat saddle points.

In these terrains, vanilla SGD oscillates wildly along the steep walls while making agonizingly slow progress along the flat valley toward the true minimum.

To solve this, modern deep learning relies on advanced optimization algorithms that incorporate two key ideas: Momentum (carrying forward directional velocity) and Adaptive Learning Rates (scaling steps per parameter based on past gradients). The standard-bearer that unifies both concepts is Adam.

1. Momentum: Accelerating Down Ravines

Vanilla SGD updates weights strictly based on the current step's gradient: $\theta_{t+1} = \theta_t - \eta g_t$. If gradients alternate signs rapidly (oscillating across a canyon), the model wastes compute moving back and forth.

Momentum borrows intuition from Newtonian physics: imagine a heavy ball rolling down a hillside. As it rolls, it builds up velocity in the consistent downhill direction and resists sharp sideways deflections.

Mathematical Formulation

At step $t$, the optimizer updates an exponentially decaying running average of past gradients—the velocity vector ($v_t$):

$$v_t = \beta \cdot v_{t-1} + (1 - \beta) \cdot g_t$$

$$\theta_{t+1} = \theta_t - \eta \cdot v_t$$

Where:

  • $g_t = \nabla_\theta J(\theta_t)$: Current gradient vector.
  • $\beta$ (beta): Momentum coefficient (typically $0.9$), dictating how much previous velocity persists.
  • $\eta$ (eta): Learning rate.
Without Momentum (Vanilla SGD)              With Momentum
      \      /      \                           \
       \    /        \                           \
        \  /          \                           \======> Accelerated Path
         \/            \                           \
    (Wasted oscillations)                   (Oscillations cancel out)