Chapter 18 of 25

Weight Initialization

Why the starting point of training matters so much

12 min read

Motivation

Imagine handing four new employees the exact same job description, the exact same desk setup, and the exact same instructions on day one — down to the letter. Every time feedback comes in from a manager, it arrives identically to all four of them, so they all update their behavior in exactly the same way, forever. A year later, you don't have four employees who've specialized into four different roles. You have one employee's behavior, copy-pasted four times, still copy-pasted four times.

That's precisely what happens when a neural network's weights all start out identical. Before a network can learn anything useful with backpropagation, its neurons need a reason to become different from one another in the first place. Getting that starting point right — or wrong — decides whether training even has a chance to work, long before the first gradient update happens.

A Real-Life Scenario

Picture a hospital deploying a network to predict patient risk from lab values, vitals, and history. The engineer, being cautious, initializes every weight to zero — reasoning that "zero" is neutral and unbiased. Training runs for hours. Loss barely moves. Every hidden neuron in the first layer, it turns out, is computing the exact same number, receiving the exact same gradient, and updating in the exact same direction as every other neuron in that layer. A hundred-neuron hidden layer is behaving like a single neuron wearing a hundred name tags.

Contrast that with a second engineer who initializes weights randomly, but with values that are far too large. Now the raw sums flowing through the network explode layer after layer, saturating every sigmoid neuron into its flat regions, where gradients are almost exactly zero. Training stalls again — for a completely different reason. Getting initialization right sits precisely between these two failure modes.

Building the Intuition

Here's the mental picture to hold onto before any formulas:

A layer of $n$ neurons is only useful if those $n$ neurons end up computing $n$ different things. If they all start identical, backpropagation gives them no way to break the tie — every neuron in that layer sees the same input, produces the same output, and receives the same gradient, so they update in lockstep and remain identical no matter how many epochs pass.
The fix is to seed randomness into the weights, so each neuron starts life computing something slightly different, giving each one a distinct gradient and a distinct path to specialize.
But randomness alone isn't enough — the scale of that randomness matters just as much as its presence. Too large, and the raw sums entering each layer balloon as they pass through the network, pushing saturating activations into flat, near-zero-gradient territory from the very first step. Too small, and signals shrink toward zero the deeper they travel, again starving the network of usable gradients.
The goal, precisely, is to pick a random distribution whose variance keeps the scale of activations (and gradients flowing backward) roughly stable as they pass through every layer — neither exploding nor vanishing.

Why the single perceptron never had this problem

Earlier, when the perceptron was introduced, its weights were initialized to zero without any issue at all. That's not a contradiction — it's because a lone perceptron has no "sibling" neurons in the same layer that need to be told apart from one another. The symmetry problem only exists when multiple neurons share a layer and would otherwise be indistinguishable. The moment you stack a hidden layer with more than one neuron, that safety net disappears.

The Symmetry-Breaking Problem, In Detail

Never initialize all weights in an MLP to zero, or to any identical constant

If every weight feeding into a hidden layer is the same number, every neuron in that layer computes an identical pre-activation from any given input. Identical pre-activations produce identical activations. Identical activations produce identical downstream contributions to the loss, which means every neuron in that layer receives an identical gradient during backpropagation. Since the update rule applies that identical gradient to already-identical weights, the weights stay identical after the update too — and this repeats every single epoch, forever. A layer of $n$ neurons collapses into the behavior of one neuron duplicated $n$ times, permanently wasting almost all of the network's representational capacity.

The fix is to initialize weights randomly, so every neuron starts by computing a slightly different function of the input, receives a slightly different gradient, and is therefore free to specialize as training proceeds. Biases, interestingly, don't need this treatment — they're commonly initialized to exactly zero, because the random weights alone are already enough to break the symmetry between neurons.

Symmetry Problem: Identical vs. Randomly Initialized NeuronsIllustration coming soon

Identical weights force every neuron in a layer to move in lockstep forever; random weights let each neuron carve out its own role during training.

Why the Scale of Randomness Matters

Simply drawing random numbers isn't the end of the story — the variance of that randomness has to be chosen deliberately.

Too large or too small both break training

If initial weights are too large, pre-activations grow larger and larger as they pass through successive layers. For saturating activations like sigmoid and tanh, this pushes neurons into their flat regions from the very first forward pass, producing vanishing gradients before training has even had a chance to begin. If initial weights are too small, activations shrink toward zero layer by layer — again starving the network of gradient signal, or leaving every neuron stuck in an almost-linear, uninformative regime.

The principled solution is to choose a random distribution whose variance is derived mathematically, so that the variance of activations — and, symmetrically, the variance of gradients flowing backward — stays approximately constant as data flows through every layer, neither growing nor shrinking layer to layer. That derivation is exactly what the following schemes do, each tailored to a different activation function.

Xavier (Glorot) Initialization

Xavier / Glorot Initialization

W_{ij} \sim \mathcal{U}\left(-\sqrt{\frac{6}{n_{in}+n_{out}}},\ \sqrt{\frac{6}{n_{in}+n_{out}}}\right) \quad \text{or equivalently} \quad W_{ij} \sim \mathcal{N}\left(0,\ \frac{2}{n_{in}+n_{out}}\right)

He (Kaiming) Initialization

He Initialization

W_{ij} \sim \mathcal{N}\left(0,\ \frac{2}{n_{in}}\right)

A favorite interview question

"Why does He initialization use a different formula from Xavier?" The clean answer: ReLU zeros out about half its inputs, which halves the forward-pass variance compared to a symmetric activation; doubling the initialization variance exactly compensates for that loss, keeping signal magnitude stable layer to layer. Xavier assumes a symmetric, non-zeroing activation like tanh or sigmoid, where no such compensation is needed.

LeCun Initialization

W_{ij} \sim \mathcal{N}\left(0,\ \frac{1}{n_{in}}\right)

Comparison at a Glance

Scheme	Variance	Recommended For
Zero / constant	0	Never for weights (symmetry problem); fine for biases
Naive random (e.g., small uniform)	Arbitrary, unscaled	Not recommended — the scale isn't principled
Xavier / Glorot	$2/(n_{in}+n_{out})$	Sigmoid, tanh
He / Kaiming	$2/n_{in}$	ReLU, Leaky ReLU, ELU
LeCun	$1/n_{in}$	SELU

Implementing It

import numpy as np
 
def xavier_init(n_in, n_out, rng):
    limit = np.sqrt(6 / (n_in + n_out))
    return rng.uniform(-limit, limit, size=(n_out, n_in))
 
def he_init(n_in, n_out, rng):
    std = np.sqrt(2 / n_in)
    return rng.normal(0, std, size=(n_out, n_in))
 
rng = np.random.default_rng(42)
W_xavier = xavier_init(n_in=128, n_out=64, rng=rng)
W_he     = he_init(n_in=128, n_out=64, rng=rng)
 
print("Xavier weights: mean=%.5f std=%.5f" % (W_xavier.mean(), W_xavier.std()))
print("He weights:     mean=%.5f std=%.5f" % (W_he.mean(), W_he.std()))
 
# Frameworks provide these directly, e.g. in PyTorch:
#   nn.init.xavier_uniform_(layer.weight)
#   nn.init.kaiming_normal_(layer.weight, nonlinearity='relu')
# and in Keras, via the `kernel_initializer` argument:
#   Dense(64, activation='relu', kernel_initializer='he_normal')

The xavier_init and he_init functions differ only in which denominator they use in the variance formula — everything else about drawing the random matrix is identical. Notice that both take n_in and n_out as explicit arguments rather than hardcoding them, since the correct variance is a function of layer shape, not a universal constant. The commented-out lines at the bottom are worth remembering for real projects: production code almost never calls these formulas by hand, since every major framework exposes them as a one-line kernel_initializer or nn.init call.

Practical rule of thumb

Use He initialization for any layer followed by a ReLU-family activation — by far the most common case in modern MLPs — and Xavier initialization for any layer followed by sigmoid or tanh. There's rarely a good reason to hand-implement either in production, but understanding the derivation is what lets you diagnose training instability that traces back to a bad initialization choice.

Key Takeaways

Zero (or any identical constant) weight initialization causes the symmetry problem: every neuron in a layer stays identical to every other neuron in that layer, forever, no matter how long training runs.
A single perceptron never faced this problem because it has no sibling neurons in its layer to distinguish from one another — the problem only appears once a layer has more than one neuron.
Weights must be initialized randomly, and the variance of that randomness is just as important as the randomness itself, to avoid vanishing or exploding activations from the very first forward pass.
Xavier/Glorot initialization ( $2/(n_{in}+n_{out})$ ) is designed for symmetric, non-zeroing activations like sigmoid and tanh.
He/Kaiming initialization ( $2/n_{in}$ ) compensates for ReLU zeroing out half its inputs and is the standard default for ReLU-family activations.
LeCun initialization ( $1/n_{in}$ ) is required for SELU's self-normalizing property to hold mathematically.
Biases are commonly initialized to zero without issue, since random weights alone already break the symmetry.

Common Mistakes

Assuming zero-initialization is always safe

Zero-initialization is fine for a single-neuron model like the perceptron, but catastrophic for any hidden layer with more than one neuron. It's an easy habit to carry over incorrectly once you start stacking layers.

Using the wrong initialization for the activation function

Applying Xavier initialization ahead of a ReLU layer (or He initialization ahead of a tanh layer) doesn't cause an outright crash, but it silently produces a mismatched signal variance, which shows up later as unexpectedly slow or unstable training that can be hard to trace back to its actual cause.

Ignoring initialization when debugging vanishing/exploding gradients

When gradients vanish or explode from the very first training step — before any optimizer quirks or architectural issues could plausibly explain it — the initialization scheme is one of the first places to look, not one of the last.

Interview Corner

Quick Check

1. Why does initializing all weights in a hidden layer to zero prevent an MLP from learning effectively?

2. Why does He initialization use double the variance of Xavier initialization for comparably sized layers?

Where This Shows Up in Practice

Every major deep learning framework defaults new layers to a sensible initialization scheme automatically — Keras's Dense layer defaults to Glorot uniform, PyTorch's nn.Linear uses a Kaiming-derived scheme by default — precisely because getting this step wrong is such a common and avoidable source of failed training runs. Understanding why these defaults exist is what lets you recognize when the default is wrong for your architecture (for instance, switching to He initialization the moment you swap a sigmoid hidden layer for ReLU), and it's exactly the kind of foundational detail that separates a model that trains smoothly from one that mysteriously never converges.