Chapter 9 of 25

Linear Algebra for Neural Networks

Vectors, matrices, and why networks are just matrix math

22 min read

Motivation

Here's a fact that surprises a lot of newcomers: a neural network with a million parameters isn't running a million separate little calculations when it makes a prediction. It's running a small handful of matrix multiplications. Every weighted sum inside a neuron, every layer's worth of neurons computed at once, every gradient flowing backward during training — all of it is linear algebra, dressed up in neural network vocabulary.

This matters for a very practical reason, not just an aesthetic one: linear algebra is why deep learning is fast. GPUs are extraordinarily good at one thing — multiplying large matrices — and the entire deep learning software stack is built around exploiting that fact. If you understand the operations in this chapter, every equation in the rest of this course will read as arithmetic you already know, instead of unfamiliar notation.

A Real-Life Scenario

Imagine a streaming service scoring how likely you are to enjoy a movie, based on a profile of features: how much you liked comedies, how much you liked action films, your average rating for movies from the last five years, and so on — say, 50 features in total. Now imagine it needs to do this not just for you, but for 10 million users, against a catalog of 20,000 movies, updated every night.

Doing this one multiplication at a time, in a Python for loop, would take unreasonably long. Doing it as one enormous matrix multiplication — every user's feature vector against every movie's feature vector, all at once — is something specialized hardware can chew through in seconds. The operations in this chapter are exactly what make that possible, and they are exactly what happens, at a smaller scale, inside every layer of an MLP.

Building the Intuition

Before the notation, here's the mental model to hold onto:

A scalar is just a single number: your age, a temperature, a loss value.
A vector is an ordered list of numbers — think of it as one row of a spreadsheet, or one data point's full set of features.
A matrix is a grid of numbers — think of it as an entire spreadsheet, or a whole batch of data points stacked together.
Inside an MLP, a layer's weights are a matrix, its bias is a vector, and a single input example is a vector. Multiplying the weight matrix by the input vector computes the pre-activation of every neuron in that layer, simultaneously — that single fact is the reason this chapter exists.

Scalar, Vector, Matrix, Tensor

a = 3.5

Vectors

Addition and Scalar Multiplication

Vector Addition and Scalar Multiplication

\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_1+v_1 \\ u_2+v_2 \\ \vdots \\ u_n+v_n \end{bmatrix}, \qquad c\mathbf{v} = \begin{bmatrix} cv_1 \\ cv_2 \\ \vdots \\ cv_n \end{bmatrix}

The Dot Product

Dot Product

\mathbf{u}\cdot\mathbf{v} = \mathbf{u}^\top\mathbf{v} = \sum_{i=1}^n u_iv_i = u_1v_1 + u_2v_2 + \cdots + u_nv_n

Worked example: let $\mathbf{w} = (0.5, -1.2, 2.0)$ and $\mathbf{x} = (1.0, 3.0, 0.5)$ .

\mathbf{w}^\top\mathbf{x} = (0.5)(1.0) + (-1.2)(3.0) + (2.0)(0.5) = 0.5 - 3.6 + 1.0 = -2.1

With bias $b = 0.3$ , the neuron's pre-activation is $z = -2.1 + 0.3 = -1.8$ — exactly the same arithmetic as a perceptron's weighted sum from two chapters ago, just written in vector notation.

Vector Norms

L1, L2, and L∞ Norms

\|\mathbf{v}\|_1 = \sum_{i=1}^n |v_i|, \qquad \|\mathbf{v}\|_2 = \sqrt{\sum_{i=1}^n v_i^2}, \qquad \|\mathbf{v}\|_\infty = \max_i |v_i|

For $\mathbf{v} = (3, -4)$ : $\|\mathbf{v}\|_1 = |3| + |-4| = 7$ , and $\|\mathbf{v}\|_2 = \sqrt{3^2+(-4)^2} = \sqrt{25} = 5$ .

Matrices

Matrix Multiplication

Matrix multiplication is what lets you compute an entire layer of neurons in one shot, instead of looping over dot products one neuron at a time.

Matrix Multiplication

\mathbf{A} \in \mathbb{R}^{m\times n}

How Matrix Multiplication Combines Rows and ColumnsIllustration coming soon

Entry C_ij is the dot product of row i of A with column j of B. For the multiplication to be valid, A's number of columns must match B's number of rows.

Worked example: let $\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 0 & -1 \end{bmatrix}$ and $\mathbf{B} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}$ .

\mathbf{AB} = \begin{bmatrix} (1)(3)+(2)(4) \\ (0)(3)+(-1)(4) \end{bmatrix} = \begin{bmatrix} 11 \\ -4 \end{bmatrix}

A Full Layer Is One Matrix Multiplication

\mathbf{W} \in \mathbb{R}^{n_{out}\times n_{in}}

Matrix multiplication is not commutative

$\mathbf{AB} \neq \mathbf{BA}$ in general — and often one of the two products isn't even defined, due to shape mismatches. Always check dimensions before multiplying; this is one of the most common bugs when implementing neural networks from scratch.

The Transpose

Matrix Transpose

(\mathbf{A}^\top)_{ij} = A_{ji}

Element-wise (Hadamard) Product

Hadamard Product

(\mathbf{A}\odot\mathbf{B})_{ij} = A_{ij}B_{ij}

Don't mix these two products up

Matrix multiplication ( $\mathbf{AB}$ ) combines rows with columns via dot products and can change the shape entirely. The Hadamard product ( $\mathbf{A}\odot\mathbf{B}$ ) requires identical shapes and multiplies entry-by-entry, preserving shape. They solve different problems and are not interchangeable.

Special Matrices Worth Recognizing

Special matrices frequently encountered in deep learning
Matrix	Description
Identity (I)	1s on the diagonal, 0s elsewhere; satisfies AI = IA = A
Zero matrix	Every entry is zero
Diagonal matrix	All off-diagonal entries are zero
Symmetric matrix	Satisfies A = Aᵀ
Orthogonal matrix	Satisfies AᵀA = I; preserves vector lengths and angles — relevant to some weight initialization schemes covered later in this course

Implementing It

import numpy as np
 
# Vectors
u = np.array([1.0, 2.0, 3.0])
v = np.array([4.0, -1.0, 0.5])
 
print("u + v      =", u + v)
print("dot(u, v)  =", np.dot(u, v))          # equivalently u @ v
print("L2 norm(u) =", np.linalg.norm(u))
print("L1 norm(u) =", np.linalg.norm(u, 1))
 
# Matrices
W = np.array([[0.2, -0.5, 1.0],
              [0.8,  0.1, -0.3]])   # shape (2, 3): 2 output units, 3 inputs
x = np.array([1.0, 0.5, -2.0])      # shape (3,)
b = np.array([0.1, -0.2])           # shape (2,)
 
z = W @ x + b                       # shape (2,): pre-activations of a layer
print("z =", z)
 
# Batch of examples: X has shape (batch_size, n_features)
X = np.array([[1.0, 0.5, -2.0],
              [0.0, 1.0,  1.0]])    # 2 examples, 3 features each
Z = X @ W.T + b                     # shape (2, 2): pre-activations for both examples
print("Z (batched) =\n", Z)

A few things worth noticing: the @ operator is NumPy's matrix multiplication, distinct from * (which does element-wise/Hadamard multiplication). Notice the batched line, X @ W.T + b: rather than looping over each of the two examples in Python, one matrix multiplication computes both examples' pre-activations at once, with b automatically broadcast across every row. This vectorized batch formulation, not manual loops, is what lets deep learning frameworks exploit fast, GPU-accelerated computation.

Remember the batched shape convention

Always compute a batch's pre-activations as X @ W.T + b, where X has shape (batch_size, n_features) and W has shape (n_outputs, n_features). Getting this transpose backwards is one of the most common shape-mismatch errors when implementing a layer from scratch.

Key Takeaways

Vectors and matrices are the basic data structures of an MLP: inputs, weights, and biases are all represented this way.
The dot product $\mathbf{w}^\top\mathbf{x}$ computes one neuron's weighted sum; matrix multiplication $\mathbf{Wx}$ computes an entire layer's pre-activations at once.
Norms ( $\|\cdot\|_1$ , $\|\cdot\|_2$ ) measure vector size and underlie regularization and gradient clipping.
Matrix multiplication requires matching inner dimensions and is not commutative — check shapes carefully.
The Hadamard (element-wise) product is a distinct operation from matrix multiplication, and it's exactly what backpropagation uses to combine gradients with activation derivatives.
NumPy's @ operator and broadcasting make all of these operations efficient and concise in code.

Common Mistakes

Confusing matrix multiplication with the Hadamard product

W @ x and W * x do very different things in NumPy, and mixing them up silently produces wrong (but sometimes shape-compatible!) results. Matrix multiplication combines rows and columns via dot products; the Hadamard product multiplies entry-by-entry and requires identical shapes.

Forgetting to check dimension compatibility before multiplying

$\mathbf{AB}$ is only defined when $\mathbf{A}$ 's column count matches $\mathbf{B}$ 's row count — and even when both $\mathbf{AB}$ and $\mathbf{BA}$ happen to be defined, they are almost never equal. Always sanity-check shapes, especially when reshaping between single-example and batched code.

Interview Corner

Quick Check

1. For weight matrix W with shape (n_out, n_in) and input vector x with shape (n_in,), what is the shape of z = Wx?

2. What is the key difference between matrix multiplication (AB) and the Hadamard product (A ⊙ B)?

Where This Shows Up in Practice

Every forward pass through every layer of an MLP is a matrix multiplication plus a bias vector; every backward pass reuses transposed weight matrices and Hadamard products with activation derivatives. The next chapter builds the other half of the toolkit — calculus — so that when we derive backpropagation, both the "what" (linear algebra, from this chapter) and the "why it works" (calculus, next) will already feel familiar.