Chapter 23 of 25
MLPs Compared to Other Architectures
When to reach for an MLP, and when not to
Motivation
Every architecture in deep learning gets invented for the same reason: someone hit a wall that the tools available at the time couldn't get past. The MLP you've spent this whole course mastering was itself one of those walls being broken — it solved the perceptron's inability to handle non-linearly-separable data. But the MLP has its own walls, and the architectures that came after it — CNNs, RNNs, LSTMs, GRUs, and Transformers — each exist because an MLP, used naively, ran straight into one of them.
This chapter isn't about learning those architectures in depth (that's for a different course). It's about being able to answer the question that separates someone who memorized backpropagation from someone who actually understands neural networks: given this problem, why would — or wouldn't — you reach for a plain MLP?
A Real-Life Scenario
Imagine you're handed three completely different jobs on the same afternoon. Job one: predict house prices from a spreadsheet of forty numeric columns (square footage, number of bedrooms, zip code, and so on). Job two: caption a photograph. Job three: translate a sentence from English to French.
Using the exact same tool for all three would be a mistake, and it wouldn't even be close. The spreadsheet job is a natural fit for an MLP — the data is already a fixed-size vector of meaningful numbers. The photograph has spatial structure a flattened vector destroys. The sentence has variable length and order that a fixed-size input can't represent at all. Recognizing which structural mismatch you're facing is exactly what lets you pick — or design — the right architecture instead of forcing a square peg into a round hole.
Building the Intuition
Here's the mental model worth carrying forward:
- An MLP assumes its input is a fixed-size vector of independent features, with no assumed relationship between one input position and its neighbors.
- Images violate that assumption spatially — a pixel's neighbors matter, and flattening the image into a vector throws that relationship away.
- Sequences (text, audio, time series) violate it temporally — element 5 depends on what came before it, and the input isn't even a fixed length.
- Every architecture beyond the MLP is best understood as: "take the MLP's general-purpose machinery, and bolt on one extra structural assumption that matches the data."
And critically — none of these newer architectures throw the MLP away. As you'll see below, the MLP survives inside almost every one of them, usually as the final layer(s) that turn extracted features into an actual answer.
The MLP's Home Turf: Perceptron vs. MLP
Before comparing the MLP to anything else, it's worth being precise about what upgrading from a single perceptron to a full MLP actually bought you.
| Aspect | Perceptron | Multi-Layer Perceptron |
|---|---|---|
| Layers | Single layer | One or more hidden layers plus an output layer |
| Activation | Step function (non-differentiable) | Smooth, differentiable nonlinearities (ReLU, sigmoid, etc.) |
| Representable functions | Only linearly separable functions | Any continuous function, given enough hidden units (Universal Approximation Theorem) |
| Training algorithm | Perceptron learning rule (error-driven, no gradient) | Backpropagation + gradient-based optimization |
| Can solve XOR? | No | Yes |
MLP vs. CNN — When Spatial Structure Matters
A Convolutional Neural Network replaces (some of) an MLP's fully-connected layers with convolutional layers: small, shared filters slid across local regions of an image. Instead of every input pixel connecting to every neuron with its own unique weight, the same small set of weights is reused at every spatial position, directly exploiting the fact that nearby pixels are related and that a cat is still a cat if it shifts three pixels to the left.
Feed a raw image straight into an MLP and two problems appear immediately. First, you have to flatten the 2D grid into a 1D vector, which throws away the fact that a pixel's neighbors are spatially meaningful — after flattening, two pixels that were touching might end up far apart in the vector. Second, the parameter count explodes: a fully-connected layer on a modest image needs over 150,000 weights per output neuron, just in the very first layer.
| Aspect | MLP | CNN |
|---|---|---|
| Connectivity | Fully connected between layers | Locally connected via shared convolutional filters |
| Parameter count for images | Very large (no weight sharing) | Much smaller (weight sharing across spatial locations) |
| Spatial structure | Not preserved (input is flattened) | Explicitly preserved and exploited |
| Typical use | Tabular data, or as a "head" after feature extraction | Images and other grid-structured data |
MLP vs. RNN — When Order and Memory Matter
An MLP expects one fixed-size input vector and treats every input independently — there's no notion of "before" or "after." A Recurrent Neural Network adds a hidden state that gets updated at every time step and carries information forward through a sequence, giving the network a form of memory an MLP architecturally cannot have.
For sequential data — text, time series, audio — where order and context genuinely change the meaning of what comes next, a plain MLP simply has nowhere to put that information. It has no mechanism for "what happened three steps ago" to influence "what happens now."
| Aspect | MLP | RNN |
|---|---|---|
| Input | Fixed-size vector | Variable-length sequence |
| Memory of past inputs | None (each input treated independently) | Yes, via a recurrent hidden state |
| Weight sharing | None across "positions" | Same weights reused at every time step |
| Typical use | Tabular data | Sequential data (text, time series, audio) |
MLP vs. LSTM/GRU — When Long-Range Memory Matters
Plain RNNs suffer badly from the vanishing gradient problem when backpropagating through many time steps, which makes learning long-range dependencies nearly impossible. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) networks fix this with learned gates — small, sigmoid-activated switches that control what information gets added to, removed from, or read out of the hidden state, specifically engineered to keep gradients flowing over long sequences.
| Aspect | MLP | LSTM / GRU |
|---|---|---|
| Sequential processing | No | Yes |
| Long-range dependency handling | Not applicable | Explicitly designed for, via gating mechanisms |
| Internal complexity | Simple: linear + activation per layer | More complex: multiple gates per cell (LSTM: input, forget, output; GRU: reset, update) |
| Relationship to MLP | N/A | Each gate is itself a small MLP-like linear-plus-sigmoid transformation |
MLP vs. Transformer — When Parallelism and Very Long Range Matter
RNNs (and LSTM/GRU) process a sequence one step at a time, which is slow — training can't be parallelized across the sequence — and still imperfect at very long ranges. Transformers throw recurrence out entirely and replace it with self-attention: a mechanism that computes relationships between every pair of positions in a sequence at once, enabling both massive parallelization and much stronger long-range modeling.
| Aspect | MLP | Transformer |
|---|---|---|
| Core mechanism | Fully connected linear layers + activation | Self-attention + position-wise feedforward (MLP) sublayers |
| Parallelizability over sequence | N/A (no sequence notion) | Fully parallelizable across sequence positions |
| Long-range dependencies | N/A | Handled directly via attention, not step-by-step recurrence |
| Relationship to MLP | N/A | Every Transformer block contains an MLP sublayer; attention itself is not an MLP |
The Vision Transformer (ViT) takes the Transformer idea and applies it to images: it splits an image into fixed-size patches, linearly embeds each one (a simple, effectively single-layer MLP projection), and feeds the resulting sequence of patch embeddings into a standard Transformer encoder — no convolutions at all. At large enough data scale, this general-purpose combination can match or beat CNNs, even though CNNs have spatial locality "hard-coded" into their design and ViTs don't.
The Big Picture: The MLP as a Universal Building Block
Across every architecture compared above, the MLP never actually disappears — it becomes a component. It typically shows up as the final classification or regression head bolted onto the end, or as the point-wise nonlinear transformation applied after some more specialized mechanism (convolution, recurrence, or attention) has already done the work of capturing structure specific to that data type.
This is the single most important takeaway of this chapter: the MLP is not a competitor to CNNs, RNNs, or Transformers — it's their common foundation and final stage. Every one of those architectures is, in a real sense, "an MLP, plus one extra idea for handling a specific kind of structure."
Comprehensive Comparison at a Glance
| Architecture | Best Suited For | Key Structural Idea |
|---|---|---|
| Perceptron | Linearly separable binary classification | Single linear decision boundary |
| MLP | Tabular data; universal component | Fully connected layers + nonlinearity |
| CNN | Images, grid-structured data | Local, weight-shared convolutional filters |
| RNN | Short sequences | Recurrent hidden state |
| LSTM / GRU | Longer sequences | Gated recurrent hidden state |
| Transformer | Long sequences, language, parallelizable training | Self-attention + MLP sublayers |
| Vision Transformer | Large-scale image tasks | Patch embedding + Transformer |
Key Takeaways
- Every architecture beyond the MLP exists to address one specific structural limitation of using a plain MLP directly on that data type: spatial structure (CNN), sequence and memory (RNN), long-range dependency (LSTM/GRU), and parallelizable long-range modeling (Transformer).
- CNNs exploit spatial locality and weight sharing; RNNs add a recurrent hidden state for memory; LSTM/GRU add gating to preserve gradients over long sequences; Transformers replace recurrence with self-attention for full parallelism.
- Despite all these specialized mechanisms, the MLP remains present as a component in nearly every one of them — most commonly as the final output head, and, in Transformers, as the position-wise feedforward sublayer inside every block.
- Understanding the MLP deeply is therefore foundational to understanding every architecture in this comparison, not a stepping stone you leave behind.
Common Mistakes
It's tempting to think "MLP vs. CNN vs. Transformer" is a single either-or choice, like picking one tool and discarding the rest. In practice, almost every modern model is a hybrid — a CNN backbone feeding an MLP classification head, a Transformer whose every block contains an MLP sublayer. The question is rarely "MLP or not," it's "where does the MLP go, and what specialized mechanism feeds it."
CNNs, RNNs, and Transformers exist to solve specific structural mismatches — they are not universally "better" than an MLP. For genuinely tabular data with no spatial or sequential structure, a plain MLP (or even a classical model like gradient-boosted trees) is often the simpler, more appropriate, and more data-efficient choice.
Interview Corner
Where This Shows Up in Practice
Recognizing which structural assumption a problem needs — spatial locality, sequential memory, long-range gating, or fully parallel attention — is exactly the skill that separates picking an architecture from memorizing one. In practice, most production systems are hybrids: a CNN or Transformer backbone extracts structure-aware features, and an MLP head turns those features into the final prediction. Knowing the MLP as thoroughly as this course does is what makes every one of those larger systems legible, rather than a black box.