Calculus for Neural Networks

Motivation

Linear algebra, from the last chapter, tells you what a neural network computes — a chain of matrix multiplications and activations. Calculus is what tells the network how to get better. Every single weight update that ever happens during training is, underneath the surface, the result of a derivative calculation. If linear algebra is the network's anatomy, calculus is its learning mechanism.

This chapter builds the calculus toolkit — derivatives, partial derivatives, gradients, and above all the chain rule — that you need to fully understand backpropagation, coming up soon. Take the chain rule section seriously: it is, without much exaggeration, the single most important piece of math in this entire course.

A Real-Life Scenario

Imagine you're adjusting the temperature dial on an oven while baking bread, trying to hit a perfect golden crust. You don't know the exact function relating dial position to crust color, but you can experiment: nudge the dial up slightly, see if the crust darkens faster or slower than before. That nudge-and-observe process — how much does the output change when I change this one input by a tiny amount? — is exactly what a derivative measures.

Now imagine that instead of one dial, you have a million dials (the weights of a neural network), and instead of eyeballing crust color, you have a single number telling you how wrong the whole network's prediction was (the loss). You can't nudge a million dials one at a time and see what happens — that would take forever. What you need is a way to instantly know, for every dial simultaneously, which direction to turn it to make the loss smaller. That's exactly what gradients and the chain rule deliver, and it's exactly what makes training feasible at all.

Building the Intuition

Here's the picture to hold before any formulas:

• A derivative tells you the instantaneous rate of change of a function — nudge the input a tiny bit, and the derivative tells you how much and in which direction the output moves.
• A partial derivative does the same thing, but for a function of many variables at once, treating every variable except one as temporarily frozen.
• The gradient collects every partial derivative into a single vector that points in the direction of steepest increase — its negative points toward the fastest way to decrease the function, which is exactly what we want when trying to shrink a loss.
• The chain rule is how you compute a derivative through a chain of nested functions — and a neural network, layer stacked on layer, is nothing but a very long chain of nested functions.

The Math Behind It

The Derivative

Partial Derivatives

A neural network's loss doesn't depend on one variable — it depends on every weight and bias simultaneously, which can number in the millions. To handle functions of many variables, we need partial derivatives.

Worked example: let $f(x,y) = 3x^2y + 2y^3$.

At $(x,y)=(1,2)$: $\frac{\partial f}{\partial x} = 6(1)(2) = 12$, and $\frac{\partial f}{\partial y} = 3(1)^2+6(2)^2 = 3+24 = 27$.

The Gradient

The Chain Rule

This is the one piece of math this entire course is quietly building toward. It is the mathematical fact that makes backpropagation possible.

A simple numeric worked example: let $y = (3x+1)^2$. Set $u = 3x+1$, so $y=u^2$.

Check by direct expansion: $y=(3x+1)^2 = 9x^2+6x+1$, so $\frac{dy}{dx}=18x+6$ — the two methods agree, exactly as they must. Concretely, at $x=2$: $u = 3(2)+1=7$, so $\frac{dy}{dx} = 6(7) = 42$, and direct expansion gives $18(2)+6=42$ too.

Now suppose a neural network has many variables feeding into the final loss through many paths at once — this is the situation in every real layer.

Gradient Descent: A First Look

Implementing It

import numpy as np
 
def f(x):
    return (3*x + 1)**2
 
def numerical_derivative(f, x, h=1e-6):
    return (f(x + h) - f(x - h)) / (2*h)
 
x0 = 2.0
analytical = 18*x0 + 6            # derived by hand above
numerical = numerical_derivative(f, x0)
 
print(f"Analytical derivative at x={x0}: {analytical}")
print(f"Numerical  derivative at x={x0}: {numerical:.6f}")

This compares a hand-derived ("analytical") derivative against a numerical approximation using tiny finite differences — nudging $x$ by a small $h$ in both directions and measuring the resulting change in $f$. The two should agree almost exactly (both give 42 here). This technique is called gradient checking, and it's an invaluable debugging tool once you implement backpropagation from scratch: it catches subtle sign errors or incorrect chain-rule applications that would otherwise silently make a network train incorrectly.

Key Takeaways

Common Mistakes

Interview Corner

Where This Shows Up in Practice

Every single training step of every neural network in existence is the chain rule, applied automatically and efficiently across every layer, computing exactly how much each weight should move to reduce the loss. Combined with the matrix and vector operations from the previous chapter, you now have both halves of the toolkit needed to derive backpropagation completely, layer by layer, symbol by symbol — which is exactly where this course goes next, right after one more chapter on probability and statistics that explains why we use the loss functions we use in the first place.