Optimizers

Motivation

Plain gradient descent works, in the sense that it's guaranteed to move downhill. But "guaranteed to eventually get there" and "gets there in a reasonable amount of time" are very different promises. Real loss surfaces have narrow ravines, wildly different sensitivities in different directions, and long, nearly flat plateaus — and plain gradient descent, armed with nothing but the current gradient and a single fixed learning rate, handles all of these badly. It oscillates in ravines, crawls across plateaus, and needs its one learning rate to somehow be right for every parameter simultaneously, which is rarely possible.

Every optimizer in this chapter exists to fix one specific weakness in the one before it. Understanding that chain of fixes — not just memorizing formulas — is what makes the whole family click into place, and it's exactly why Adam, the optimizer nearly everyone reaches for by default, looks the way it does.

A Real-Life Scenario

Picture someone hiking down a steep, narrow canyon toward a river at the bottom. If they only ever step in the exact direction the ground slopes right under their feet, they'll zigzag wildly back and forth across the narrow canyon walls, making painfully slow net progress toward the river even though every single step is technically "downhill." A more experienced hiker instead keeps some momentum from their recent direction of travel, smoothing out the zigzag into a more direct path forward — and if they also start favoring paths where the ground has been gently sloped before (rather than steep and jagged), they get down even faster and with much less wasted motion.

That's the whole story of this chapter: momentum smooths the zigzag, and adaptive per-parameter learning rates favor the gentle, informative directions over the steep, jittery ones.

Building the Intuition

• Every optimizer here builds on the same skeleton: compute a gradient, then adjust the parameters by some function of that gradient. The differences are all in what extra information gets tracked and used.
• Momentum-based methods remember a running average of past gradients, smoothing out noisy zigzagging.
• Adaptive methods (AdaGrad, RMSProp, Adam) give every individual parameter its own effective learning rate, based on how large that parameter's gradients have historically been.
• Adam simply combines both ideas — momentum's smoothing and RMSProp's per-parameter adaptivity — into one optimizer, which is why it dominates as the default choice today.

The Math Behind It

Batch, Stochastic, and Mini-Batch Gradient Descent

Momentum

Nesterov Accelerated Gradient (NAG)

Standard momentum computes the gradient at the current position, then applies the accumulated velocity. Nesterov momentum instead computes the gradient at the position the velocity is about to move to — a "look-ahead" step — giving the optimizer a chance to correct course before overshooting rather than after. This typically yields faster, more stable convergence than plain momentum.

AdaGrad

RMSProp

RMSProp fixes AdaGrad's core problem by replacing the ever-growing sum $G_t$ with an exponentially decaying moving average $\mathbf{s}_t$ (with decay controlled by $\beta$, typically 0.9). Because old gradients are gradually "forgotten," the effective learning rate no longer shrinks all the way to zero, and training can keep adapting throughout the entire run.

AdaDelta

AdaDelta extends RMSProp one step further by also maintaining a moving average of past parameter updates (not just gradients), and uses the ratio of these two moving averages to size each update. This removes the need to manually specify a global learning rate $\eta$ at all — a nice property in theory, though in practice Adam has become far more popular thanks to its strong empirical performance and simpler, more intuitive hyperparameters.

Adam (Adaptive Moment Estimation)

Adam combines Momentum's idea (tracking a moving average of gradients) with RMSProp's idea (tracking a moving average of squared gradients) into one optimizer — currently the default choice for training most modern neural networks, including MLPs.

AdamW

Standard Adam, when combined with L2 regularization implemented as an added penalty term inside the loss, interacts poorly with Adam's adaptive per-parameter learning rates — effectively weakening the regularization for parameters with large accumulated gradients. AdamW fixes this by applying weight decay as a separate, direct step ($\theta_t \leftarrow \theta_t - \eta\lambda\theta_{t-1}$) rather than folding it into the gradient computation, decoupling it from Adam's adaptive scaling. AdamW is now the standard choice in most modern deep learning practice, particularly for Transformer-based models.

Nadam and Lion (Brief Notes)

Nadam (Nesterov-accelerated Adam) applies the same look-ahead idea used in Nesterov momentum to Adam's momentum term, aiming to combine Adam's adaptive learning rates with Nesterov's improved convergence behavior. It typically performs comparably to, or slightly better than, standard Adam.

Lion ("EvoLved Sign Momentum") is a more recent optimizer, discovered via automated search over optimizer designs, that uses only the sign of a momentum-smoothed gradient to update parameters — rather than the gradient's raw magnitude — combined with decoupled weight decay similar to AdamW. It uses noticeably less memory than Adam, since it doesn't track a second moment, and has shown strong results on some large-scale vision and language models, though it's less thoroughly battle-tested than Adam/AdamW across the full breadth of deep learning tasks.

Interactive: Watch Learning Rate Choice Play Out

The optimizers in this chapter all still depend on getting the learning rate roughly right — momentum and adaptivity smooth the path, but they don't rescue a wildly mis-scaled step size. Revisit the same bowl-shaped descent from earlier in this course and see convergence, divergence, and oscillation play out directly.

Comparison Table

Implementing It

import numpy as np
 
def L(theta): return theta[0]**2 + 10*theta[1]**2
def grad_L(theta): return np.array([2*theta[0], 20*theta[1]])
 
def run_sgd(steps=50, lr=0.05):
    theta = np.array([5.0, 5.0])
    for _ in range(steps):
        theta -= lr * grad_L(theta)
    return theta
 
def run_momentum(steps=50, lr=0.05, beta=0.9):
    theta = np.array([5.0, 5.0]); v = np.zeros(2)
    for _ in range(steps):
        g = grad_L(theta)
        v = beta*v + (1-beta)*g
        theta -= lr * v
    return theta
 
def run_adam(steps=50, lr=0.1, b1=0.9, b2=0.999, eps=1e-8):
    theta = np.array([5.0, 5.0]); m = np.zeros(2); v = np.zeros(2)
    for t in range(1, steps+1):
        g = grad_L(theta)
        m = b1*m + (1-b1)*g
        v = b2*v + (1-b2)*g**2
        m_hat = m / (1 - b1**t)
        v_hat = v / (1 - b2**t)
        theta -= lr * m_hat / (np.sqrt(v_hat) + eps)
    return theta
 
print("Final theta (SGD):     ", run_sgd())
print("Final theta (Momentum):", run_momentum())
print("Final theta (Adam):    ", run_adam())
print("True minimum: [0. 0.]")

The toy loss $L(\theta) = \theta_0^2 + 10\theta_1^2$ is deliberately shaped like a narrow ravine — the second coordinate has 10x the curvature of the first — which is exactly the kind of loss surface where plain SGD zigzags and momentum-based or adaptive methods visibly do better. Try running all three and comparing how many steps each needs to get close to [0, 0].

Key Takeaways

Common Mistakes

Interview Corner

Where This Shows Up in Practice

Nearly every deep learning framework ships Adam or AdamW as a one-line default optimizer, and for good reason — the chain of ideas in this chapter (smoothing via momentum, adaptivity via RMSProp-style scaling, stability via bias correction) addresses the real, practical failure modes of plain gradient descent on the messy, non-convex loss surfaces described earlier in this course. Choosing an optimizer is rarely the deciding factor between a good and a great model, but choosing a badly mismatched one can easily prevent training from working at all.