Glossary

This glossary collects every key term introduced across the course — from the foundational ideas of machine learning, through the mathematical building blocks, to the vocabulary of training and tuning a multi-layer perceptron — in one alphabetized reference you can search instead of hunting back through chapters.

A

Adam Optimizer

An optimizer that combines the smoothing effect of momentum with per-parameter adaptive learning rates, plus a bias-correction step that keeps its early updates well-behaved. It's the most common default choice for training neural networks in practice.

Artificial Intelligence

The broad field concerned with building machines that can perform tasks normally thought to require human intelligence. Machine learning, and everything else in this course, is one approach within that larger field.

Artificial Neuron

The basic computational unit of a neural network: it takes a set of inputs, multiplies each by a weight, adds a bias, sums the result, and passes that sum through a nonlinear activation function.

B

Backpropagation

The algorithm that efficiently computes the gradient of the loss with respect to every weight in a multi-layer network, by applying the chain rule and working backward from the output layer to the input layer.

Batch Normalization

A technique that normalizes the inputs to a layer using the statistics of the current mini-batch, then rescales and shifts the result using two learnable parameters. It stabilizes and speeds up training by keeping layer inputs in a consistent range.

Bias-Variance Tradeoff

The balance between a model being too simple to capture the true pattern in the data (high bias, underfitting) and being too sensitive to the quirks of its specific training sample (high variance, overfitting). Most modeling decisions ultimately trade a little of one for less of the other.

C

Chain Rule

The calculus rule for differentiating a function that's built by composing other functions. It's the mathematical foundation that makes backpropagation possible, since a neural network is just one long composition of functions.

Convex Function

A function whose graph never dips below a straight line drawn between any two of its points. Optimizing a convex function is comparatively easy, because it's guaranteed to have exactly one global minimum and no misleading local ones.

Cross-Entropy Loss

A loss function that measures how different a predicted probability distribution is from the true label distribution. It arises naturally from Maximum Likelihood Estimation and is the standard loss for classification problems.

D

Deep Learning

A subfield of machine learning built on multi-layer neural networks, which learn hierarchical, increasingly abstract representations of the data automatically, layer by layer.

Dot Product

The sum of the products of two vectors' corresponding entries. It's the exact operation a neuron uses to combine its inputs and weights into a single weighted sum.

Dropout

A regularization technique that randomly switches off a fraction of neurons during each training step, forcing the network to avoid relying too heavily on any single neuron and improving generalization.

Dying ReLU

A failure mode in which a ReLU neuron's pre-activation drifts permanently negative, so its output and gradient become stuck at zero — the neuron effectively stops learning for the rest of training.

E

Early Stopping

A regularization strategy that halts training as soon as performance on a validation set stops improving, rather than training for a fixed number of epochs regardless of what's happening to generalization.

Error Signal (δ)

The partial derivative of the loss with respect to a given layer's pre-activation. It's the central quantity backpropagation computes at every layer, and every weight gradient falls out of it directly.

Expectation

The long-run average value a random variable would take if you could observe it over and over, weighted by how likely each outcome is.

F

Forward Propagation

The process of computing a network's prediction by passing an input through each layer in sequence, from the input layer to the output.

G

Gradient

The vector made up of every partial derivative of a function, one for each input variable. It points in the direction where the function increases fastest — which is exactly why gradient descent moves in the opposite direction to reduce a loss.

Gradient Descent

An iterative optimization algorithm that repeatedly nudges a model's parameters in the direction opposite to the loss function's gradient, gradually reducing the loss.

H

He Initialization

A weight initialization scheme tuned for ReLU-family activation functions, designed to keep the scale of activations and gradients stable as they pass through many layers.

Hyperparameter

Any configuration choice — such as learning rate, batch size, or number of hidden layers — that's set by the practitioner before training begins, rather than learned automatically from the data.

L

Learning Rate

A hyperparameter that controls how large a step gradient descent takes at each update. Too small and training crawls; too large and it can overshoot or diverge entirely.

Linear Separability

A property of a dataset where its two classes can be perfectly divided by a single straight line (or, more generally, a hyperplane). It's the exact condition a perceptron needs in order to succeed.

M

Machine Learning

A subfield of AI in which a system improves its performance on a task by learning from data and experience, instead of following rules a human wrote out explicitly.

Matrix

A rectangular grid of numbers, used throughout this course to represent linear transformations — most importantly, the full set of weights connecting one layer of a network to the next.

Maximum Likelihood Estimation (MLE)

A method for choosing model parameters by finding the values that make the observed training data as probable as possible under the model. It's the theoretical justification behind why cross-entropy loss is the natural choice for classification.

Multi-Layer Perceptron (MLP)

A feedforward neural network built from one or more hidden layers plus an output layer, trained end-to-end using backpropagation.

N

Norm

A function that measures the size or length of a vector. The L1 and L2 norms are the two most common versions used throughout machine learning, including in regularization.

O

Overfitting

What happens when a model fits its training data — including the noise and quirks specific to that sample — so closely that it fails to generalize well to new, unseen data.

P

Perceptron

The earliest trainable model of an artificial neuron: a linear binary classifier trained using the perceptron learning rule, which updates its weights only when it makes a mistake.

S

Saddle Point

A point on a function's surface where it curves upward in some directions and downward in others, rather than being a clean minimum or maximum. These are common obstacles in the non-convex, high-dimensional loss surfaces neural networks are trained on.

Softmax

An activation function that converts a vector of raw scores (logits) into a valid probability distribution across multiple classes, ensuring every output is positive and all of them sum to one.

Supervised Learning

A machine learning paradigm where a model learns a mapping from inputs to outputs using a training set of labeled input-output pairs.

U

Underfitting

What happens when a model is too simple to capture the real underlying pattern in the data, leading to poor performance even on the training set itself.

Universal Approximation Theorem

The theorem stating that a large enough single-hidden-layer network can, in principle, approximate any continuous function on a bounded input domain. It's a guarantee about what could exist, not about whether training will actually find it.

V

Vanishing Gradient Problem

The tendency for gradients to shrink exponentially smaller as they're backpropagated through many layers that use saturating activation functions, which stalls learning in the network's earliest layers.

Variance

A measure of how spread out a random variable's possible values are around its expected value.

Vector

An ordered list of numbers, used throughout this course to represent a single point, direction, or set of features in space.

W

Weight Decay

The direct, multiplicative shrinking of weights toward zero at every update step. Under plain stochastic gradient descent, it's mathematically equivalent to L2 regularization.

X

Xavier Initialization

A weight initialization scheme tuned for sigmoid and tanh activation functions, designed to keep the variance of activations and gradients consistent as they flow through the network.

XOR Problem

The classic example of a function whose classes cannot be separated by any single straight line, making it the textbook demonstration of exactly what a single-layer perceptron cannot learn.