Optimization Theory

Motivation

Imagine you're standing somewhere on a foggy mountain range at night, and your only goal is to reach the lowest valley. You can't see more than a few feet in any direction. All you can do is feel the slope under your feet, take a step downhill, feel again, and repeat. That's it. That's the entire strategy.

That blind, step-by-step descent is not a metaphor we're borrowing loosely — it is, almost exactly, what happens every time a neural network trains. "Training a model" sounds like a mysterious, almost magical process, but underneath the vocabulary it is one very concrete thing: searching for the set of numbers (weights and biases) that makes a loss function as small as possible. Every neuron, every layer, every activation choice you've studied so far only matters because, eventually, something has to go find good values for all of it. This chapter is about that "something."

A Real-Life Scenario

Picture a delivery company trying to minimize the total distance its trucks drive every day. There's no formula you can solve once and be done — the "cost" (total distance) depends on hundreds of interacting decisions (which truck goes where, in what order), and the landscape of possible costs is lumpy, full of better and worse regions, with no obvious shortcut to the best one. The company's planners nudge the routes a little, check whether total distance went down, nudge again, and keep repeating.

A neural network's training loop is the same story, except the "routes" are millions of weights, and the "total distance" is the loss function. Instead of a human planner nudging things by hand, gradient descent does the nudging automatically, guided by calculus instead of intuition.

Building the Intuition

Before touching a single equation, hold onto three ideas:

• Training is search, not calculation. There's no closed-form formula that spits out the best weights directly — the network has to iteratively try, measure, and adjust.
• The loss function is the landscape. Every possible setting of the weights is a point on this landscape, and the height at that point is how bad the network's predictions are. Training means walking downhill on this landscape.
• The landscape's shape matters enormously. A landscape shaped like a smooth bowl is easy to descend — go downhill anywhere, and you eventually reach the one and only bottom. A landscape full of dips, ridges, and flat plateaus is a much harder walk, and this second kind of landscape is exactly what a real neural network produces.

The Math Behind It

Training as an Optimization Problem

Convex vs. Non-Convex Landscapes

Critical Points: Minima, Maxima, and Saddle Points

A critical point is any point $\theta$ where $\nabla L(\theta) = \mathbf{0}$ — the landscape is momentarily flat in every direction. Not all flat spots are the same:

Batch, Stochastic, and Mini-Batch Gradient Descent

The basic update rule is $\theta \leftarrow \theta - \eta\nabla_\theta L(\theta)$, but there's a choice hiding inside it: how much data do you use to compute that gradient before taking a step?

The full comparison — including code and how these interact with modern optimizers like Adam — is the subject of the next chapter. For now, the key idea is simply that "gradient descent" is a family, not a single algorithm.

The Role of the Learning Rate

Worked Example: Gradient Descent by Hand

Minimize $L(\theta) = \theta^2 - 4\theta + 5$, starting at $\theta_0 = 0$ with learning rate $\eta = 0.3$.

Step 1 — the gradient: $L'(\theta) = 2\theta - 4$.

Iteration 1: $\theta_0 = 0 \implies L'(0) = -4 \implies \theta_1 = 0 - 0.3(-4) = 1.2$

Iteration 2: $\theta_1 = 1.2 \implies L'(1.2) = -1.6 \implies \theta_2 = 1.2 - 0.3(-1.6) = 1.68$

Iteration 3: $\theta_2 = 1.68 \implies L'(1.68) = -0.64 \implies \theta_3 = 1.68 - 0.3(-0.64) = 1.872$

Each step's correction shrinks as $\theta$ approaches the minimum — that's the gradient itself shrinking, since $L'(\theta) = 2\theta - 4$ approaches zero as $\theta \to 2$. Continuing this process, $\theta$ converges toward $\theta^{*}=2$, which checks out algebraically: $L'(\theta) = 0 \implies \theta = 2$, and $L''(\theta) = 2 > 0$ confirms it's a minimum, not a maximum.

Implementing It

import numpy as np
 
def L(theta):
    return theta**2 - 4*theta + 5
 
def grad_L(theta):
    return 2*theta - 4
 
theta = 0.0
lr = 0.3
history = [theta]
 
for step in range(15):
    g = grad_L(theta)
    theta = theta - lr * g
    history.append(theta)
 
print("Trajectory of theta:", [round(t, 4) for t in history])
print("Final theta (should approach 2.0):", round(theta, 4))
print("Final loss (should approach 1.0):", round(L(theta), 4))

This mirrors the hand-traced example exactly: each iteration recomputes the gradient at the current point and takes a proportional step against it. Running it confirms theta converges to approximately 2.0, matching the algebraic solution.

Key Takeaways

Common Mistakes

Interview Corner

Where This Shows Up in Practice

Every training run you'll ever launch — whether it's a two-layer MLP or a billion-parameter language model — is an instance of this exact search problem. The specific algorithm used to conduct that search (plain gradient descent, Momentum, Adam, and beyond) is the subject of the next two chapters, but the underlying goal never changes: find $\theta$ that makes $L(\theta)$ as small as possible, one step at a time, guided by the gradient.