Backpropagation

Motivation

Suppose a group project gets a failing grade. The professor doesn't just shrug — they need to figure out exactly which teammate's section pulled the score down, and by how much, so each person knows precisely what to fix next time. Do it fairly and efficiently, and the whole group improves fast. Do it by guessing, and you're stuck.

A multi-layer perceptron faces the exact same problem after every single prediction. The network gets a loss — a single number saying "you were this wrong" — and it needs to know how much every individual weight, in every layer contributed to that wrongness. With potentially millions of weights spread across many layers, this is called the credit assignment problem, and backpropagation is the algorithm that solves it, efficiently and exactly, using nothing more exotic than the chain rule from calculus.

A Real-Life Scenario

Think of a factory assembly line producing a final product, where the final inspector rejects it for being 2cm too long. That error didn't originate at the final station — it could have started at station 1, been slightly amplified at station 2, and slightly corrected at station 3. To fix the process, you need to trace the error back through every station and figure out each one's share of the blame.

Backpropagation does exactly this inside a neural network. It starts with the error at the output (the loss), and walks backward through every layer, computing exactly how much each layer — and each weight within that layer — is responsible for the final mistake.

Building the Intuition

Here's the picture to hold in your head before any formulas:

• Forward pass: information flows left to right — input → layer 1 → layer 2 → ... → output → loss.
• Backward pass: blame flows right to left — loss → output layer → ... → layer 2 → layer 1.
• At every layer, backprop computes one central quantity: how sensitive is the final loss to a tiny change in this layer's raw output? That sensitivity is called the layer's error signal, written $\boldsymbol{\delta}^{[\ell]}$.
• Once you know a layer's error signal, computing that layer's weight gradients is simple arithmetic — no calculus left to do. All the hard work is in propagating $\boldsymbol{\delta}$ backward correctly.

The Math Behind It

We reuse the forward-propagation notation: $\mathbf{z}^{[\ell]} = \mathbf{W}^{[\ell]}\mathbf{a}^{[\ell-1]}+\mathbf{b}^{[\ell]}$, $\mathbf{a}^{[\ell]} = \varphi^{[\ell]}(\mathbf{z}^{[\ell]})$, and a per-example loss $L = \ell(\hat{\mathbf{y}}, \mathbf{y})$ that can be any differentiable loss function.

Step 1 — The Output Layer Error

By the chain rule, the error at the very last layer $L$ splits into two ingredients:

Step 2 — Propagating the Error Backward

This is the heart of the algorithm: relating layer $\ell$'s error to the error one layer ahead of it.

Where it comes from: layer $\ell$'s pre-activation only affects the loss through layer $\ell+1$. Summing over every neuron in layer $\ell+1$ that a given neuron feeds into (the multivariate chain rule) gives:

which, written across every neuron $j$ at once, is exactly the matrix recursion above.

Step 3 — Turning Error Signals Into Gradients

Once every $\boldsymbol{\delta}^{[\ell]}$ is known, the actual trainable parameters fall out directly — no more chain rule required.

Worked Example: Backpropagation Through a 2-Layer Network

We continue the network from Chapter 13's worked example: input $\mathbf{x}=(1.0, 0.5)$, sigmoid activations throughout, with the forward pass already computed as $\mathbf{z}^{[1]}=(0.30,0.45)$, $\mathbf{a}^{[1]}=(0.5744,0.6106)$, $z^{[2]}=0.2931$, and $\hat{y} = 0.5728$. The true label is $y = 1$, and we use binary cross-entropy with a sigmoid output — so, using the simplification above:

Layer 2 gradients:

Backward recursion into layer 1, using $\sigma'(z)=\sigma(z)(1-\sigma(z)) \approx (0.2445,\ 0.2378)$:

Layer 1 gradients:

Every gradient needed for one step of gradient descent is now available — computed with only local quantities and a single backward pass.

Implementing It

import numpy as np
 
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
 
def sigmoid_deriv(a):
    return a * (1 - a)   # takes the ACTIVATION a = sigmoid(z), not z
 
x = np.array([1.0, 0.5]); y = 1.0
W1 = np.array([[0.3, -0.2], [0.5, 0.1]]); b1 = np.array([0.1, -0.1])
W2 = np.array([[0.8, -0.6]]);             b2 = np.array([0.2])
 
# --- Forward pass ---
z1 = W1 @ x + b1
a1 = sigmoid(z1)
z2 = W2 @ a1 + b2
a2 = sigmoid(z2)          # y_hat
 
# --- Backward pass ---
delta2 = a2 - y                                   # BCE + sigmoid simplification
dW2 = np.outer(delta2, a1)
db2 = delta2
 
delta1 = (W2.T @ delta2) * sigmoid_deriv(a1)
dW1 = np.outer(delta1, x)
db1 = delta1

The comment on sigmoid_deriv matters more than it looks: that function takes the activation $a=\sigma(z)$, not the pre-activation $z$, because $\sigma'(z) = \sigma(z)(1-\sigma(z))$ can be written directly in terms of the already-computed $a$. This is exactly why the forward pass caches every $\mathbf{z}^{[\ell]}$ and $\mathbf{a}^{[\ell]}$ — backprop reuses them instead of recomputing anything.

Key Takeaways

Common Mistakes

Interview Corner

Where This Shows Up in Practice

Every deep learning framework — PyTorch, TensorFlow, JAX — is built around an automatic differentiation engine that is, at its core, backpropagation generalized to arbitrary computation graphs. Understanding the derivation in this chapter by hand is what makes concepts like vanishing gradients (Chapter 15), gradient clipping, and modern optimizers (Chapter 17) make sense — they're all strategies for keeping this exact backward flow of error signals well-behaved.