Chapter 19 of 25

Regularization

Keeping a network from memorizing instead of learning

19 min read

Motivation

A student who memorizes last year's exact exam questions and answers can ace that exact test perfectly — and then fail miserably the moment this year's exam asks the same concepts in a slightly different way. They didn't learn the subject; they learned the answer key. A neural network can fall into precisely the same trap: given enough capacity, it can memorize the training set — including its noise, its quirks, its coincidences — well enough to get training loss arbitrarily close to zero, while learning almost nothing that generalizes to new data.

Regularization is the umbrella term for every technique that fights this tendency — that nudges a network toward learning genuine patterns instead of memorizing specific examples. Because MLPs, especially deep and wide ones, have enormous raw capacity to memorize, regularization isn't an optional extra; it's used in nearly every practical training run.

A Real-Life Scenario

Imagine a hospital training a model to predict patient risk from lab results. With enough hidden units, the network is powerful enough to essentially assign a unique internal "fingerprint" to each training patient and simply memorize their outcome — including quirks that have nothing to do with actual risk, like a particular lab machine's rounding behavior on a particular day. On the training set, this model looks flawless. Deployed on new patients next month, it performs barely better than a coin flip, because it never learned the actual medical relationship between lab values and risk — it learned a lookup table.

Regularization is what keeps that hospital's model honest: constraining it just enough that memorizing noise becomes a losing strategy, while learning the real underlying signal remains a winning one.

Building the Intuition

Before the formulas, here's the shared theme running through every technique in this chapter:

Overfitting happens when a model has more freedom than the genuine signal in the data requires, so it spends that extra freedom fitting noise.
Nearly every regularization technique works by taking away some of that freedom — penalizing large weights, randomly disabling parts of the network, stopping training before it has time to memorize, or injecting a bit of noise into training itself.
None of these techniques change what the network is fundamentally capable of representing — they change what it's encouraged to actually learn, given a fixed amount of training data and time.

L1 and L2 Regularization

The most direct way to limit a network's freedom is to penalize the weights themselves, right inside the loss function.

L2 Regularization (Weight Decay)

L_{\text{total}} = L_{\text{data}} + \frac{\lambda}{2}\sum_\ell\|\mathbf{W}^{[\ell]}\|_2^2

L1 Regularization

L_{\text{total}} = L_{\text{data}} + \lambda\sum_\ell\|\mathbf{W}^{[\ell]}\|_1

Elastic Net: getting both effects at once

Elastic Net combines both penalties: $L_{\text{total}} = L_{\text{data}} + \lambda_1\sum\|\mathbf{W}\|_1 + \lambda_2\sum\|\mathbf{W}\|_2^2$ , aiming for L1's sparsity-inducing behavior alongside L2's smoother, more stable shrinkage.

L1 vs. L2 regularization
Aspect	L1	L2
Effect on weights	Drives many weights to exactly zero (sparsity)	Shrinks all weights smoothly, rarely to exactly zero
Interpretation	Implicit feature selection	Smooth complexity control
Gradient	Constant magnitude (±λ)	Proportional to weight magnitude
Typical use	When a sparse, interpretable model is desired	The default, general-purpose choice for MLPs

Dropout

Definition

Dropout is a regularization technique that, during each training step, randomly "drops" (sets to zero) each neuron's activation with probability $p$ (typically 0.2 to 0.5), independently for each neuron and each training example.

Dropout (Inverted Dropout, as used in practice)

\mathbf{a}^{[\ell]}_{\text{dropout}} = \frac{\mathbf{m} \odot \mathbf{a}^{[\ell]}}{1-p}, \qquad m_i \sim \text{Bernoulli}(1-p)

Why dropout works — two complementary intuitions

(1) Preventing co-adaptation: without dropout, neurons can become overly reliant on the specific presence of particular other neurons, developing brittle, co-dependent features. Dropout forces each neuron to remain useful across many different random subsets of the other neurons, encouraging more robust, individually meaningful features.

(2) Implicit ensembling: training with dropout can be viewed as simultaneously training an enormous number of different "thinned" sub-networks — each random mask defines a different sub-network — that share weights, with the final trained network approximating an average over all of them. Ensembles are well known to generalize better than any single model, and dropout gets much of that benefit essentially for free.

Dropout: Random Neuron Deactivation During TrainingIllustration coming soon

Each training step randomly zeroes a different subset of neurons, forcing the network to avoid relying too heavily on any single one.

The most common dropout bug

Forgetting to disable dropout at inference/evaluation time (or forgetting the $1/(1-p)$ rescaling during training) produces a systematic mismatch between training-time and test-time activation magnitudes, silently degrading model performance. Frameworks handle this automatically via a model.eval() (PyTorch) or training=False (Keras) mode flag — always double-check it's set correctly before evaluation.

Early Stopping

Definition

Early stopping monitors a model's performance on a held-out validation set during training, and stops training once validation performance stops improving for a specified number of epochs (the "patience"), even if training loss is still decreasing.

This directly targets the classic overfitting signature: training loss keeps dropping while validation loss starts climbing back up. That crossover point marks where the network switches from learning genuine patterns to memorizing training-specific noise.

Always restore the best checkpoint, not the last one

Save the model checkpoint corresponding to the best validation performance observed so far during training — not simply whatever the model looks like at the moment training stops. Most frameworks' early-stopping callbacks support automatically restoring the best-performing checkpoint, and it's worth always enabling that option.

Batch Normalization

Definition

Batch Normalization (BatchNorm) is a layer, typically inserted between a layer's linear transformation and its activation function, that normalizes its input to have approximately zero mean and unit variance across the current mini-batch, then applies a learnable scale $\gamma$ and shift $\beta$ .

Batch Normalization

z_1, \dots, z_B

Why the learnable γ and β are necessary

Normalizing to zero mean and unit variance without any learnable adjustment would forcibly constrain every neuron's pre-activation distribution, which can actually hurt representational power — a sigmoid neuron, for instance, might specifically benefit from a wider input range to fully use its non-saturated region. The learnable $\gamma,\beta$ let the network learn the optimal scale and shift for each neuron, with strict standardization available as a special case ( $\gamma=1,\beta=0$ ) if that genuinely is optimal.

Why Batch Normalization helps train deep networks:

It keeps pre-activation distributions stable across layers and across training steps, mitigating vanishing/exploding gradients and allowing safely higher learning rates.
It provides a mild regularizing effect, because the batch statistics $\mu_B,\sigma_B^2$ are noisy estimates that vary from batch to batch — acting somewhat like noise injection, similar in spirit to dropout, though through a different mechanism.
In practice it very often makes training faster and more stable, and makes networks noticeably less sensitive to weight initialization choices.

BatchNorm needs different behavior at inference

At inference time, mini-batches are often unavailable, or would make predictions depend unpredictably on whichever other examples happen to share the batch. Instead, BatchNorm uses a running (exponentially averaged) estimate of $\mu,\sigma^2$ accumulated during training, fixed at inference time. Forgetting to switch a framework's BatchNorm layer into evaluation mode before inference is a very common and easy-to-miss bug — the same distinction that trips people up with dropout.

Comparison at a Glance

Technique	Mechanism
L1 / L2 / Elastic Net	Penalize large weights directly in the loss function
Dropout	Randomly zero out neurons during training to prevent co-adaptation
Early Stopping	Stop training when validation performance stops improving
Batch Normalization	Normalize layer inputs per mini-batch; stabilizes and mildly regularizes training
Weight Decay	Equivalent to L2 regularization when implemented as direct multiplicative shrinkage (as in AdamW)

Implementing It

from tensorflow import keras
from tensorflow.keras import layers
 
model = keras.Sequential([
    layers.Input(shape=(20,)),
    layers.Dense(64, use_bias=False),      # bias omitted: BatchNorm has its own shift (beta)
    layers.BatchNormalization(),
    layers.Activation("relu"),
    layers.Dropout(0.3),                   # drop 30% of neurons during training
    layers.Dense(64, activation="relu"),
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])
 
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
model.summary()

Notice use_bias=False on the first Dense layer — when a layer is immediately followed by Batch Normalization, its bias term is redundant, since BatchNorm's learnable $\beta$ already supplies a shift. Dropping it saves a small number of unnecessary parameters. The two Dropout(0.3) layers each independently zero out 30% of their inputs during training only; Keras automatically switches this off during evaluation and prediction.

When a layer feeds into BatchNorm

Set use_bias=False on that layer, as shown above — it's a small but common practical detail worth remembering, since a redundant bias parameter is otherwise silently trained for no benefit.

Key Takeaways

Regularization keeps a network from memorizing training data — including its noise — instead of learning genuinely generalizable patterns.
L2 regularization (weight decay) smoothly shrinks all weights; L1 regularization induces sparsity by driving many weights exactly to zero.
Dropout randomly zeroes neurons during training, preventing co-adaptation and approximating an ensemble of sub-networks; it must be disabled at inference.
Early stopping halts training based on validation performance, directly targeting the point where a model starts overfitting.
Batch Normalization standardizes layer inputs per mini-batch (with learnable scale/shift), stabilizing and often accelerating training, with a mild regularizing side effect.
Dropout and BatchNorm both require switching between distinct training and inference behavior — a frequent source of bugs when forgotten.

Common Mistakes

Forgetting to switch modes between training and inference

Both dropout and BatchNorm behave differently during training versus inference. Evaluating a model without switching it into evaluation mode silently produces incorrect, inconsistent results — one of the most common practical bugs when implementing these from scratch.

Treating regularization strength as a fixed constant

$\lambda$ (for L1/L2) and $p$ (for dropout) are hyperparameters that need tuning for each problem, not universal constants. Too strong, and the network underfits by being over-constrained; too weak, and overfitting returns.

Confusing BatchNorm's regularizing side effect with its main purpose

BatchNorm was originally motivated by stabilizing training dynamics, not by regularizing — its mild regularization effect is a helpful side benefit, not a substitute for dropout or weight decay when overfitting is severe.

Interview Corner

Quick Check

1. Why does L1 regularization tend to produce sparse weight matrices while L2 does not?

2. Why must dropout and batch normalization behave differently during training versus inference?

Where This Shows Up in Practice

Virtually every production MLP combines several of these techniques at once — weight decay via an optimizer like AdamW, dropout between hidden layers, batch normalization for training stability, and early stopping to decide when to stop, all running simultaneously. None of them is a silver bullet in isolation; the real skill in practice is choosing the right combination and strength for a given dataset size and model capacity, which is exactly the kind of judgment call the next chapter's full training pipeline is built to support.