Learn ML

Chapter 11 of 25

Probability & Statistics Essentials

Making sense of uncertainty in data and predictions

14 min read

Motivation

Here's a question worth sitting with: why do we train classifiers with "cross-entropy loss" and regressors with "mean squared error"? Most courses just hand you these formulas and say "use this one for classification, that one for regression," as if they were arbitrary conventions. They aren't. Both losses fall directly out of one unifying idea from probability theory, and once you see where they come from, you'll never again think of a loss function as an arbitrary design choice — you'll see it as a direct mathematical consequence of a modeling assumption you already made.

Probability and statistics are the language in which we express uncertainty — and every prediction a neural network makes is, whether we say so explicitly or not, secretly a statement about probability. A "90% confident it's spam" isn't just a nice user-facing number; it's the entire mathematical foundation the network was trained on.

A Real-Life Scenario

Think about a weather forecasting model predicting "70% chance of rain tomorrow." It isn't claiming certainty — it's expressing a calibrated belief under uncertainty, based on patterns in temperature, pressure, humidity, and historical outcomes. If it rains on 70 out of 100 days it predicted "70% chance of rain," the model is well-calibrated; if it only rains on 30 of those days, something is badly miscalibrated, even if the model still "gets the yes/no right" most of the time.

A neural network binary classifier works exactly the same way. Its sigmoid output isn't just an arbitrary number squeezed between 0 and 1 — it's explicitly interpreted as a probability, and the entire training procedure is built around making that probability as trustworthy as possible given the data. Understanding why requires exactly the tools in this chapter.

Building the Intuition

Before the formulas, here's the mental model:

  • A random variable is just a variable whose value comes from some random process — a coin flip, tomorrow's rainfall, whether an email is spam.
  • A probability distribution describes how likely each possible value is.
  • Expectation is the long-run average value you'd see if you repeated the random process many times; variance is how spread out those values tend to be around that average.
  • Maximum Likelihood Estimation asks: given the data we actually observed, which model parameters would have made that data most probable? This one question, taken seriously, is where cross-entropy loss comes from.

The Math Behind It

Three Distributions You'll See Constantly

Bernoulli Distribution
P(X=1)=p,P(X=0)=1−pP(X=1) = p, \qquad P(X=0) = 1-p

Where: p∈[0,1]p \in [0,1] is the probability of "success" (outcome 1).

Why this matters: the output of a binary classifier's sigmoid activation is interpreted as exactly this pp — the probability an example belongs to class 1. Binary cross-entropy loss, covered below, is derived directly from the Bernoulli likelihood.

Categorical Distribution
P(X=k)=pk,where ∑k=1Kpk=1, pk≥0P(X=k) = p_k, \qquad \text{where } \sum_{k=1}^K p_k = 1,\ p_k \geq 0

Why this matters: this is exactly what a softmax output layer produces — a vector of KK probabilities, one per class, that sum to 1. Categorical cross-entropy loss is derived from this distribution the same way binary cross-entropy is derived from the Bernoulli.

Gaussian (Normal) Distribution
p(x)=12πσ2exp⁡(−(x−μ)22σ2)p(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

Where: μ\mu is the mean (center) and σ2\sigma^2 is the variance (spread).

Why this matters: weight initialization schemes you'll meet later in this course draw initial weights from a Gaussian with a carefully chosen variance. The Gaussian distribution is also exactly what mean squared error loss assumes about regression noise, as the next section shows.

Gaussian Distributions With Different VariancesIllustration coming soon
Two Gaussian probability density functions with the same mean but different variances. A larger variance produces a wider, flatter distribution — the same shape used to model regression noise and to draw initial network weights.

Expectation and Variance

Expectation and Variance
E[X]=∑xx P(X=x) (discrete),E[X]=∫x p(x) dx (continuous)\mathbb{E}[X] = \sum_x x\,P(X=x) \ \text{(discrete)}, \qquad \mathbb{E}[X] = \int x\,p(x)\,dx \ \text{(continuous)}Var(X)=E[(X−E[X])2]=E[X2]−(E[X])2\text{Var}(X) = \mathbb{E}\big[(X-\mathbb{E}[X])^2\big] = \mathbb{E}[X^2] - (\mathbb{E}[X])^2

Intuition: E[X]\mathbb{E}[X] is the long-run average value of XX across many repetitions; Var(X)\text{Var}(X) measures how spread out XX's values typically are around that average.

Why this matters: expectation and variance are central to understanding batch normalization, a technique you'll meet later that normalizes each layer's activations to have approximately zero mean and unit variance.

Descriptive Statistics From Real Data

Given a finite dataset x1,…,xNx_1, \dots, x_N, we estimate the population mean and variance with sample statistics:

Sample Mean and Variance
xˉ=1N∑i=1Nxi,s2=1N∑i=1N(xi−xˉ)2\bar{x} = \frac{1}{N}\sum_{i=1}^N x_i, \qquad s^2 = \frac{1}{N}\sum_{i=1}^N (x_i - \bar{x})^2

Why this matters: these exact statistics are computed, per feature, during feature standardization (a common preprocessing step), and per batch, inside batch normalization layers.

Worked example: for a feature with values x=(2,4,4,6)x = (2, 4, 4, 6):

xˉ=2+4+4+64=4,s2=(2−4)2+(4−4)2+(4−4)2+(6−4)24=4+0+0+44=2\bar{x} = \frac{2+4+4+6}{4} = 4, \qquad s^2 = \frac{(2-4)^2+(4-4)^2+(4-4)^2+(6-4)^2}{4} = \frac{4+0+0+4}{4} = 2

The standardized values zi=xi−xˉs2z_i = \frac{x_i-\bar{x}}{\sqrt{s^2}} are (−22,0,0,22)≈(−1.41,0,0,1.41)\left(\frac{-2}{\sqrt2}, 0, 0, \frac{2}{\sqrt2}\right) \approx (-1.41, 0, 0, 1.41) — mean 0, variance 1, exactly the goal of standardizing a feature before training.

Maximum Likelihood Estimation: Where Loss Functions Actually Come From

This is the payoff of the whole chapter — the answer to why we use the specific loss functions we use.

Maximum Likelihood Estimation

Given a model that defines a probability distribution over possible outputs pθ(y∣x)p_\theta(y\mid\mathbf{x}), and a training dataset, Maximum Likelihood Estimation chooses the parameters θ\theta that maximize the probability the model assigns to the data we actually observed.

Intuition: among all the possible settings of θ\theta, pick the one under which the training data you actually collected would have been the least surprising outcome.

Deriving binary cross-entropy from scratch. Suppose the model outputs y^=pθ(y=1∣x)\hat{y} = p_\theta(y=1\mid\mathbf{x}), interpreted as a Bernoulli probability. Given NN independent training examples, the likelihood of the entire training set is:

L(θ)=∏i=1Ny^i yi(1−y^i)1−yi\mathcal{L}(\theta) = \prod_{i=1}^N \hat{y}_i^{\,y_i}(1-\hat{y}_i)^{1-y_i}

Multiplying together many small probabilities underflows to zero numerically, so instead we maximize the log-likelihood — the logarithm is monotonically increasing, so it has the same maximizer:

log⁡L(θ)=∑i=1N[yilog⁡y^i+(1−yi)log⁡(1−y^i)]\log\mathcal{L}(\theta) = \sum_{i=1}^N \Big[y_i\log\hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\Big]

Finally, since optimization is conventionally framed as minimizing a loss, we negate this and average over NN examples — which produces exactly binary cross-entropy:

L(θ)=−1N∑i=1N[yilog⁡y^i+(1−yi)log⁡(1−y^i)]L(\theta) = -\frac{1}{N}\sum_{i=1}^N \Big[y_i\log\hat{y}_i + (1-y_i)\log(1-\hat{y}_i)\Big]
The big picture

Minimizing cross-entropy loss is mathematically identical to maximizing the likelihood of the training data under a Bernoulli (or Categorical, for multi-class problems) model. This is not a coincidence or an arbitrary convention — cross-entropy is precisely the loss function that MLE derives, given the assumption that the model's output is a genuine probability. The same style of derivation, starting from a Gaussian noise assumption instead of a Bernoulli one, produces mean squared error for regression — you'll see this given full treatment in a dedicated loss-functions chapter later in this course.

A frequently asked graduate-level question

"Derive cross-entropy loss from first principles." Reproduce the steps above: start from the Bernoulli likelihood of the whole dataset, take its logarithm, negate it, and average. Being able to walk through this from memory demonstrates real understanding, not just formula recall.

Implementing It

import numpy as np
 
# Sample mean and variance, matching the worked example above
x = np.array([2.0, 4.0, 4.0, 6.0])
mean = x.mean()
var = x.var()          # population variance (divides by N, not N-1)
z = (x - mean) / np.sqrt(var)
 
print("mean =", mean, "variance =", var)
print("standardized =", z)
 
# Binary cross-entropy loss for a small batch of predictions
y_true = np.array([1, 0, 1, 1])
y_pred = np.array([0.9, 0.2, 0.6, 0.8])   # sigmoid outputs, interpreted as probabilities
 
bce = -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))
print("binary cross-entropy =", bce)

The standardization block reproduces the worked example by hand, confirming numpy's mean() and var() match the manual arithmetic above. The cross-entropy block implements the formula derived in this chapter directly: for each example, it takes −log⁡(y^)-\log(\hat y) when the true label is 1 and −log⁡(1−y^)-\log(1-\hat y) when it's 0, then averages across the batch — exactly the negated, averaged log-likelihood we derived.

Watch out for log(0)

If a predicted probability y^\hat y is ever exactly 0 or 1, np.log will produce -inf or a warning, and the loss can become nan. Production implementations clip predictions to a small epsilon away from 0 and 1 (e.g. np.clip(y_pred, 1e-7, 1-1e-7)) before taking the logarithm.

Key Takeaways

Key Takeaways
  • The Bernoulli and Categorical distributions model binary and multi-class classification outputs respectively; the Gaussian distribution underlies weight initialization and mean squared error loss.
  • Expectation measures a distribution's average value; variance measures its spread around that average.
  • Sample mean and variance (computed from real data) are used in feature standardization and inside batch normalization layers.
  • Cross-entropy loss is not an arbitrary choice — it is the direct mathematical consequence of Maximum Likelihood Estimation under a Bernoulli or Categorical output assumption.
  • The same MLE reasoning, applied to a Gaussian noise assumption instead, is what produces mean squared error loss for regression.

Common Mistakes

Treating cross-entropy and MSE as arbitrary conventions

It's easy to memorize "cross-entropy for classification, MSE for regression" as an unexplained rule of thumb. Both are consequences of a specific probabilistic assumption about the output (Bernoulli/Categorical vs. Gaussian) combined with Maximum Likelihood Estimation — understanding this lets you reason about which loss to use for distributions that don't fit either mold.

Confusing population variance with sample variance

Dividing by NN (population variance) versus N−1N-1 (the unbiased sample variance estimator, Bessel's correction) gives slightly different answers. NumPy's .var() defaults to dividing by NN unless you pass ddof=1. This rarely matters for large batches but is a common source of small numerical discrepancies.

Interview Corner

Quick Check
1. What does Maximum Likelihood Estimation choose the model's parameters to do?
2. Why do we take the logarithm of the likelihood before optimizing, rather than optimizing the likelihood directly?

Where This Shows Up in Practice

Every classifier's confidence score, every use of batch normalization, and every choice of loss function in this entire course traces back to the ideas in this chapter. With vectors and matrices (describing structure), calculus (describing how to learn), and probability (describing what a loss function actually means) all in place, you now have the complete mathematical foundation needed to derive backpropagation rigorously, layer by layer — exactly where the course goes next.