Loss Functions

Motivation

A network can't improve at something it can't measure. Before any gradient can be computed, before backpropagation has anything to propagate, there has to be a single number that says "here's exactly how wrong you were." That number is the loss, and the function that produces it — the loss function — is one of the most consequential design decisions in the entire training pipeline. Pick the wrong one, and even a perfectly implemented network trains against the wrong goal.

This chapter catalogs the loss functions actually used in practice, and — because it was explicitly promised back in the Backpropagation chapter — carefully derives why two of the most common pairings (sigmoid with binary cross-entropy, and softmax with categorical cross-entropy) collapse the output error signal down to the remarkably clean $\hat{y} - y$.

A Real-Life Scenario

Imagine grading two very different kinds of exams. On a numeric exam (predict a temperature, predict a stock price), "how wrong" is naturally a matter of degree — being off by 2 degrees is worse than being off by 0.5 degrees, and the penalty should scale smoothly with the size of the miss. That's the world of regression losses.

On a multiple-choice exam (is this email spam or not, is this a cat or a dog or a bird), "how wrong" is a different kind of question — you're not off by some numeric amount, you assigned a certain confidence to the correct answer, and the penalty should depend on how confident you were and whether you were right. That's the world of classification losses. Loss functions formalize both of these grading philosophies mathematically, and this chapter walks through the main ones on each side.

Building the Intuition

• A loss function isn't arbitrary — most of the losses in this chapter can be derived from a Maximum Likelihood Estimation argument: assume a specific noise distribution for how the true labels are generated, and the "obviously correct" loss falls straight out of the math.
• For regression, the size and shape of the error matters: squaring it punishes big mistakes heavily; taking the absolute value treats all mistakes proportionally; a hybrid can get the best of both.
• For classification, the loss cares about confidence, not just correctness: being 51% sure of the right answer is barely rewarded over being 51% sure of the wrong one, while being 99% confidently wrong is punished severely.

The Math Behind It: Regression Losses

Mean Squared Error (MSE)

Advantages: smooth, convex in $\hat y$, heavily penalizes large errors due to squaring, simple gradient. Disadvantages: very sensitive to outliers — one badly wrong prediction, squared, can dominate the entire loss. Use case: the default regression loss when errors are roughly Gaussian and outliers aren't a major concern.

Mean Absolute Error (MAE)

MAE corresponds to MLE under a Laplace noise assumption instead of Gaussian. Since errors aren't squared, large errors aren't disproportionately amplified — MAE is robust to outliers. The trade-off: its gradient has constant magnitude regardless of error size, which can make fine convergence near the minimum slower and less stable, and it's non-differentiable exactly at zero error.

Huber Loss

Huber loss combines the best of both worlds: it behaves like MSE (smooth, quadratic) for small errors, and like MAE (robust, linear) for large errors, avoiding both MSE's outlier sensitivity and MAE's kink at zero. It's common in reinforcement learning value-function training, and any regression setting needing a balance between smoothness and robustness.

The Math Behind It: Classification Losses

Binary Cross-Entropy (BCE)

Deriving the Sigmoid + BCE Simplification

This is the derivation the Backpropagation chapter promised: proving that $\delta^{[L]} = \hat y - y$ when a sigmoid output is paired with BCE loss.

For a single example, $L = -\big[y\log\hat y + (1-y)\log(1-\hat y)\big]$ where $\hat y = \sigma(z)$. First, differentiate the loss with respect to the prediction itself:

Then apply the chain rule through the sigmoid, using $\sigma'(z) = \hat y(1-\hat y)$ (derived in the Activation Functions chapter):

Use case: binary classification, always paired with a sigmoid output activation.

Categorical Cross-Entropy (CCE)

By an entirely analogous derivation to the BCE case above — using the softmax Jacobian in place of the sigmoid derivative — the softmax + CCE pairing produces the identical clean simplification $\boldsymbol{\delta}^{[L]} = \hat{\mathbf{y}} - \mathbf{y}$. This is exactly why softmax and categorical cross-entropy are always used together in practice: the same elegant cancellation that made sigmoid + BCE so clean reappears here in vector form.

Hinge Loss

Hinge loss is zero whenever the prediction has the correct sign and a margin of at least 1 from the decision boundary ($y_i\hat y_i \geq 1$); it grows linearly once the prediction crosses into incorrect or under-confident territory. This encourages not just correct classification but a confident margin around the boundary. It's historically the loss function of Support Vector Machines, and is occasionally used with MLPs for margin-based binary classification, though cross-entropy is far more common in practice.

Worked Numerical Example: BCE vs. MSE

Suppose $\hat y = 0.8$ and the true label is $y=1$:

Now suppose the model is badly wrong, $\hat y = 0.01$ for true label $y=1$:

Notice how BCE penalizes the confidently-wrong prediction far more heavily (4.605 vs. 0.223 — a roughly 20-fold increase) than MSE does (0.980 vs. 0.04 — a 25-fold increase, but on a much smaller and less discriminative absolute scale). This is exactly why cross-entropy is generally preferred for classification: it produces much sharper gradients precisely for confidently wrong predictions, driving faster correction.

Implementing It

import numpy as np
 
def mse(y, y_hat):
    return np.mean((y - y_hat)**2)
 
def mae(y, y_hat):
    return np.mean(np.abs(y - y_hat))
 
def huber(y, y_hat, delta=1.0):
    e = y - y_hat
    is_small = np.abs(e) <= delta
    small_loss = 0.5 * e**2
    large_loss = delta * (np.abs(e) - 0.5*delta)
    return np.mean(np.where(is_small, small_loss, large_loss))
 
def binary_cross_entropy(y, y_hat, eps=1e-12):
    y_hat = np.clip(y_hat, eps, 1-eps)   # avoid log(0)
    return -np.mean(y*np.log(y_hat) + (1-y)*np.log(1-y_hat))
 
def categorical_cross_entropy(y_onehot, y_hat, eps=1e-12):
    y_hat = np.clip(y_hat, eps, 1.0)
    return -np.mean(np.sum(y_onehot * np.log(y_hat), axis=-1))
 
y = np.array([1.0, 0.0, 1.0])
y_hat = np.array([0.9, 0.2, 0.6])
print("MSE :", mse(y, y_hat))
print("MAE :", mae(y, y_hat))
print("BCE :", binary_cross_entropy(y, y_hat))

The eps clipping inside binary_cross_entropy and categorical_cross_entropy isn't decorative — it directly prevents the failure mode explained below.

Comparison Table

Key Takeaways

Common Mistakes

Interview Corner

Where This Shows Up in Practice

Every training loop you'll ever write starts by picking a loss function, and that choice ripples through the entire rest of the pipeline: it determines what output activation is valid (sigmoid pairs with BCE, softmax pairs with CCE, linear pairs with MSE/MAE/Huber), and it determines exactly what gradient backpropagation starts with at the output layer. The clean $\hat y - y$ simplification derived in this chapter isn't just a mathematical curiosity — it's the literal first line of code in the backward pass of nearly every classification network in production today.