Activation Functions

Motivation

You already proved something important in the last chapter: stack as many linear layers as you like, and you still only ever get a straight line (or flat plane) at the end. The thing that breaks that curse — the single ingredient that lets depth actually buy you something — is the small nonlinear function squeezed into every neuron, right after the weighted sum. That function is called the activation function, and this chapter is a complete tour of the ones actually used in practice, including why some of them fell out of favor and why others became the default almost everywhere.

A Real-Life Scenario

Think about how a thermostat works versus how a smart climate system works. A dumb thermostat is essentially a step function: below the target temperature, heater fully on; above it, fully off. That's simple, but abrupt and uninformative — it gives you no sense of how close you are to the target, and a control system built entirely out of such switches would behave jerkily and be almost impossible to fine-tune.

A smart system instead responds smoothly: it ramps the heater up gradually as the room gets colder, and eases off gradually as it warms. That smooth, graded response is what most modern activation functions try to achieve inside a neural network — a way of responding to input that isn't just "fully on" or "fully off," but carries a continuous, informative gradient the network can learn from.

Building the Intuition

• Every activation function takes a raw weighted sum ($z$) and squashes, reshapes, or filters it in some nonlinear way before passing it to the next layer.
• A good activation function should be nonlinear (or depth adds nothing, as proven last chapter), (almost everywhere) differentiable (so backpropagation has a gradient to work with), cheap to compute (since it runs millions of times per training step), and ideally avoid saturating — having a near-zero derivative over large stretches of input.
• That last property, saturation, turns out to be the central plot of this entire chapter. It's the reason sigmoid and tanh, once the default choice, were largely replaced by ReLU and its descendants.

The Vanishing and Exploding Gradient Problem

During backpropagation, the error signal $\boldsymbol{\delta}^{[\ell]}$ gets multiplied by $\varphi'(\mathbf{z}^{[\ell]})$ at every single layer as it propagates backward. If $\varphi'$ is consistently much smaller than 1 (as with sigmoid and tanh once they saturate), these repeated multiplications shrink the gradient exponentially the deeper it travels — the vanishing gradient problem. Early layers of a deep network then receive almost no learning signal at all and effectively stop improving. The mirror-image failure, where $\varphi'$ is consistently larger than 1, causes gradients to blow up exponentially instead — the exploding gradient problem.

The Math Behind It: A Catalog of Activation Functions

Binary Step (Historical)

This is what the original perceptron used. Its gradient is zero almost everywhere, which makes it completely unusable with gradient-based learning — it survives only as a historical reference point.

Sigmoid (Logistic)

Range: $(0,1)$. Advantages: smooth, directly interpretable as a probability, historically important. Disadvantages: saturates for large $|z|$ — its maximum derivative is only $0.25$, at $z=0$ — causing severe vanishing gradients in deep networks; its output is also not zero-centered, which can slow convergence. Use case: the output layer for binary classification (interpreting the output as $P(y=1|\mathbf{x})$); rarely used in hidden layers of modern deep networks.

Hyperbolic Tangent (Tanh)

Range: $(-1,1)$. Advantages: zero-centered (unlike sigmoid), generally improving convergence speed, with a steeper gradient near zero. Disadvantages: still saturates for large $|z|$, so it still suffers vanishing gradients in deep networks. Use case: historically common in hidden layers before ReLU became standard; still seen in some recurrent architectures.

ReLU (Rectified Linear Unit)

Range: $[0,\infty)$. Advantages: trivially cheap to compute; its derivative is exactly 1 for every positive input, so it doesn't saturate on the positive side, largely fixing vanishing gradients there; empirically converges much faster than sigmoid or tanh. Disadvantages: the dying ReLU problem — a neuron whose pre-activation stays persistently negative during training receives zero gradient forever and effectively stops learning, permanently; also not zero-centered. Use case: the default choice for hidden layers in most modern feedforward and convolutional networks.

Leaky ReLU

where $\alpha$ is a small constant, typically $0.01$. This fixes the dying ReLU problem by allowing a small, nonzero gradient when $z<0$, at the cost of one extra hyperparameter to choose. It's a common drop-in replacement for ReLU when dying neurons are observed.

ELU (Exponential Linear Unit)

ELU is smooth (unlike Leaky ReLU's kink at zero), and its negative values push the mean activation closer to zero, which can speed up learning — a motivation similar to batch normalization. The trade-off is extra computational cost from the exponential.

SELU (Scaled ELU)

with fixed constants $\lambda \approx 1.0507$ and $\alpha \approx 1.6733$ (not hyperparameters to tune). Under specific conditions — LeCun normal weight initialization and a plain feedforward architecture without other normalization layers — SELU causes each layer's outputs to automatically converge toward zero mean and unit variance, a property called self-normalization. This benefit is fragile: it can break down in the presence of skip connections, standard dropout, or batch normalization.

Softplus

A smooth approximation of ReLU with no sharp kink at zero, always differentiable, but more expensive to compute and still capable of producing small gradients for very negative inputs.

Swish

where $\beta$ is a constant or learnable parameter, often simply fixed at 1. Swish is smooth and non-monotonic — it dips slightly below zero for small negative $z$ before returning to zero — and has been shown empirically to outperform ReLU on some deep architectures, at extra computational cost.

GELU (Gaussian Error Linear Unit)

where $\Phi(z)$ is the standard normal cumulative distribution function, often approximated as $0.5z\big(1+\tanh[\sqrt{2/\pi}(z + 0.044715z^3)]\big)$ for efficiency. GELU can be interpreted as weighting the input by the probability that a standard normal variable is less than $z$, and has become the standard activation in Transformer-based architectures.

Interactive: Explore Each Function and Its Derivative

Softmax (Multi-Class Output Activation)

Worked example: let $\mathbf{z} = (2.0, 1.0, 0.1)$. Then $e^{2.0}\approx7.389$, $e^{1.0}\approx2.718$, $e^{0.1}\approx1.105$, summing to about $11.212$, giving $\text{softmax}(\mathbf{z}) \approx (0.659,\ 0.242,\ 0.099)$. The class with the largest logit (2.0) receives the highest probability, and all three sum to exactly 1. Softmax is used exclusively in the output layer for multi-class, single-label classification, always paired with categorical cross-entropy loss.

Summary Comparison

Implementing It

import numpy as np
 
def sigmoid(z): return 1/(1+np.exp(-z))
def tanh(z): return np.tanh(z)
def relu(z): return np.maximum(0, z)
def leaky_relu(z, alpha=0.01): return np.where(z >= 0, z, alpha*z)
def elu(z, alpha=1.0): return np.where(z >= 0, z, alpha*(np.exp(z)-1))
def softmax(z):
    z = z - np.max(z)             # numerical stability
    e = np.exp(z)
    return e / np.sum(e)
 
z = np.array([-2.0, -0.5, 0.0, 0.5, 2.0])
print("sigmoid:   ", sigmoid(z))
print("tanh:      ", tanh(z))
print("relu:      ", relu(z))
print("leaky_relu:", leaky_relu(z))
print("elu:       ", elu(z))
print("softmax([2.0, 1.0, 0.1]):", softmax(np.array([2.0, 1.0, 0.1])))

Each function is implemented with vectorized NumPy operations so it applies element-wise across an entire array at once — exactly how it's used inside a real network, applied to every neuron in a layer simultaneously. Note the subtraction of the max value inside softmax before exponentiating; this is the numerical stability trick described above.

Key Takeaways

Common Mistakes

Interview Corner

Where This Shows Up in Practice

Every modern architecture is defined partly by its choice of activation function: convolutional networks and most feedforward MLPs default to ReLU or a variant, Transformers default to GELU, and every classification network's output layer uses sigmoid or softmax depending on whether it's binary or multi-class. The vanishing gradient problem introduced here also motivates architectural techniques covered later in this course, like batch normalization and careful weight initialization — both are, at their core, strategies for keeping activations away from the saturating regions described in this chapter.