Chapter 21 of 25

Practical Implementation

Building an MLP for real, bugs and all

17 min read

Motivation

Reading about forward passes, backpropagation, and training loops is one thing. Actually opening an editor, importing a library, and getting a real network to train on real data is a different skill entirely — and it's the one that separates understanding a concept from being able to use it. The good news is that every piece of theory covered so far maps directly onto a small number of lines of code in any modern framework. The bad news is that each framework expresses those same ideas with different defaults, different amounts of hand-holding, and different opportunities to get something subtly wrong.

This chapter closes that gap by implementing the exact same task three separate times — with scikit-learn, TensorFlow/Keras, and PyTorch — so you can see, side by side, how the abstractions differ while the underlying mathematics stays identical.

A Real-Life Scenario

Picture a hospital's data science team building a model to flag whether a tumor is likely malignant based on measured cell features — a real, high-stakes binary classification problem on tabular data, similar in spirit to the well-known Breast Cancer Wisconsin dataset. A junior team member could reach for scikit-learn and have a working baseline in ten lines. A team preparing to deploy the model at scale on a GPU cluster might reach for Keras. A research team trying an unusual custom architecture might need PyTorch's full flexibility. All three are solving the same problem — the choice of framework is a practical engineering decision, not a mathematical one.

Task Setup

We'll use a binary classification task on tabular data — predicting whether a tumor is malignant from measured numeric features — with all input features standardized to zero mean and unit variance, exactly as the training pipeline chapter recommends.

Implementation 1: scikit-learn

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score, classification_report
 
# 1. Load and split data
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
 
# 2. Standardize features (fit ONLY on training data)
scaler = StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test  = scaler.transform(X_test)
 
# 3. Define and train the MLP
model = MLPClassifier(
    hidden_layer_sizes=(64, 32),     # two hidden layers
    activation='relu',
    solver='adam',
    alpha=1e-4,                      # L2 regularization strength
    learning_rate_init=1e-3,
    batch_size=32,
    max_iter=200,
    early_stopping=True,
    validation_fraction=0.15,
    random_state=42,
)
model.fit(X_train, y_train)
 
# 4. Evaluate
y_pred = model.predict(X_test)
print("Test accuracy:", accuracy_score(y_test, y_pred))
print(classification_report(y_test, y_pred))

Notice how much is implicit here: hidden_layer_sizes=(64, 32) fully defines the architecture without ever writing a matrix shape by hand, alpha is the L2 regularization strength discussed earlier, and early_stopping=True with validation_fraction=0.15 silently carves out and monitors a validation split for you. There is no visible training loop at all — .fit() runs the entire pipeline from the previous chapter internally.

Practical notes: scikit-learn's MLPClassifier is the simplest way to get a working MLP with minimal code, ideal for tabular data and quick baselines. It hides the training loop entirely and offers less flexibility than Keras or PyTorch — no custom architectures, and critically, no GPU support.

Implementation 2: TensorFlow / Keras

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
 
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
 
# Build the model
model = keras.Sequential([
    layers.Input(shape=(X_train.shape[1],)),
    layers.Dense(64, use_bias=False, kernel_initializer="he_normal"),
    layers.BatchNormalization(),
    layers.Activation("relu"),
    layers.Dropout(0.3),
    layers.Dense(32, activation="relu", kernel_initializer="he_normal"),
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])
 
model.compile(
    optimizer=keras.optimizers.Adam(learning_rate=1e-3),
    loss="binary_crossentropy",
    metrics=["accuracy"],
)
 
early_stop = keras.callbacks.EarlyStopping(
    monitor="val_loss", patience=10, restore_best_weights=True)
 
history = model.fit(
    X_train, y_train,
    validation_split=0.15,
    epochs=100,
    batch_size=32,
    callbacks=[early_stop],
    verbose=0,
)
 
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"Test accuracy: {test_acc:.4f}")

Every line here traces back to an earlier chapter: kernel_initializer="he_normal" is the He initialization scheme paired with ReLU, BatchNormalization() and Dropout(0.3) are the regularization techniques covered previously, and the EarlyStopping callback with restore_best_weights=True implements exactly the best-checkpoint recommendation from that same chapter. Notice use_bias=False on the first Dense layer — redundant once BatchNormalization supplies its own learnable shift.

Practical notes: Keras's Sequential/Functional API mirrors this course's layer-by-layer notation almost directly: each Dense layer corresponds to one weight-and-bias pair. Keras handles backpropagation automatically via automatic differentiation — you never write a single gradient formula.

Implementation 3: PyTorch

import torch
import torch.nn as nn
import torch.optim as optim
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
 
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)
 
X_train_t = torch.tensor(X_train, dtype=torch.float32)
y_train_t = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
X_test_t  = torch.tensor(X_test,  dtype=torch.float32)
y_test_t  = torch.tensor(y_test,  dtype=torch.float32).unsqueeze(1)
 
class MLP(nn.Module):
    def __init__(self, n_in):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_in, 64, bias=False),
            nn.BatchNorm1d(64),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(32, 1),                         # raw logit (no sigmoid here)
        )
        # He initialization
        for m in self.net:
            if isinstance(m, nn.Linear):
                nn.init.kaiming_normal_(m.weight, nonlinearity='relu')
 
    def forward(self, x):
        return self.net(x)
 
model = MLP(n_in=X_train.shape[1])
criterion = nn.BCEWithLogitsLoss()                  # sigmoid + BCE, fused
optimizer = optim.Adam(model.parameters(), lr=1e-3)
 
best_val_loss, patience, patience_ctr = float("inf"), 10, 0
n_train = int(0.85 * len(X_train_t))
X_tr, X_val = X_train_t[:n_train], X_train_t[n_train:]
y_tr, y_val = y_train_t[:n_train], y_train_t[n_train:]
 
for epoch in range(100):
    model.train()
    optimizer.zero_grad()
    logits = model(X_tr)
    loss = criterion(logits, y_tr)
    loss.backward()
    optimizer.step()
 
    model.eval()
    with torch.no_grad():
        val_loss = criterion(model(X_val), y_val).item()
 
    if val_loss < best_val_loss:
        best_val_loss, patience_ctr = val_loss, 0
        best_state = {k: v.clone() for k, v in model.state_dict().items()}
    else:
        patience_ctr += 1
        if patience_ctr >= patience:
            break
 
model.load_state_dict(best_state)
model.eval()
with torch.no_grad():
    test_preds = (torch.sigmoid(model(X_test_t)) > 0.5).float()
    test_acc = (test_preds == y_test_t).float().mean().item()
print(f"Test accuracy: {test_acc:.4f}")

This version writes out, by hand, everything Keras did implicitly. The for epoch in range(100) loop is a direct, literal implementation of the training-loop algorithm from the previous chapter. model.train() and model.eval() are the explicit mode switches that dropout and batch normalization both depend on — miss either one and training-vs-inference behavior silently diverges. loss.backward() performs exactly the backpropagation algorithm derived earlier via automatic differentiation, rather than hand-written gradient formulas. The manual patience_ctr block, saving best_state every time validation improves, is early stopping implemented from first principles instead of hidden behind a callback.

Why BCEWithLogitsLoss instead of sigmoid + BCELoss

BCEWithLogitsLoss combines the sigmoid activation and binary cross-entropy loss into a single, numerically stable operation, computed directly on the raw logit rather than a separately-computed probability. This is exactly why the network's final Linear layer above has no activation — the loss function applies the sigmoid internally, in a way that avoids the numerical instability of computing $\log(\sigma(z))$ as two separate floating-point steps when $z$ is very negative or very positive. It also implements the same $\hat{y}-y$ gradient simplification proven when backpropagation was first derived.

Practical notes: PyTorch requires writing the training loop explicitly, making every step of the training pipeline directly visible in code. This is more verbose, but it's also exactly why PyTorch is often preferred for research and custom architectures — there's no hidden behavior to work around when you need something the standard API doesn't anticipate.

Framework Comparison

scikit-learn vs. Keras vs. PyTorch for MLPs
Aspect	scikit-learn	Keras/TensorFlow	PyTorch
Ease of use	Highest (a few lines)	High (declarative layers)	Moderate (explicit training loop)
Flexibility	Low (fixed MLP structure)	High	Highest
GPU support	No	Yes	Yes
Best for	Quick baselines, tabular data, teaching	Production deployment, rapid prototyping	Research, custom architectures, fine-grained control

Choosing a framework in practice

For learning purposes, implementing an MLP's forward and backward pass completely from scratch with NumPy is the single best way to deeply understand what these frameworks are doing automatically underneath their APIs. For real projects: reach for scikit-learn for quick tabular baselines, and Keras or PyTorch for anything requiring GPUs, custom architectures, or production deployment.

Same MLP, Three FrameworksIllustration coming soon

Different levels of abstraction, identical underlying mathematics: forward propagation, backpropagation via autodiff, and the same optimizer update rule.

Key Takeaways

scikit-learn's MLPClassifier offers the fastest path to a working baseline with minimal code, at the cost of flexibility and GPU support.
Keras expresses architecture declaratively, closely mirroring the layer-by-layer weight-and-bias notation used throughout this course.
PyTorch requires an explicit training loop, making every concept — forward pass, loss, backward pass, optimizer step, early stopping — directly visible and modifiable in code.
All three frameworks implement the exact same underlying mathematics: forward propagation, backpropagation via automatic differentiation, the chosen loss function, and the chosen optimizer.
use_bias=False before BatchNorm, and BCEWithLogitsLoss instead of separate sigmoid + BCE, are two small but recurring practical details worth remembering.

Common Mistakes

Forgetting model.eval() in PyTorch

Skipping model.eval() before validation or test-time inference leaves dropout active and BatchNorm using batch statistics instead of running averages, silently corrupting evaluation results. Always pair it with model.train() before resuming training.

Applying sigmoid before BCELoss instead of using the fused loss

Manually applying a sigmoid activation and then calling a plain BCE loss is both less numerically stable and easy to get subtly wrong compared to using a fused loss like BCEWithLogitsLoss, which computes the same thing more safely in one step.

Fitting the scaler after splitting incorrectly, or not at all

All three implementations above fit StandardScaler only on X_train and reuse it to transform X_test. Fitting a fresh scaler on the test set separately (or fitting on the combined data) reintroduces the data leakage problem discussed in the training pipeline chapter.

Interview Corner

Quick Check

1. In the Keras example, why is use_bias=False used on the first Dense layer?

2. In the PyTorch example, what does loss.backward() actually do?

Where This Shows Up in Practice

Nearly every production tabular-data model at a company starts life as one of these three implementations, often scikit-learn first as a quick sanity-check baseline, then Keras or PyTorch once GPU acceleration or a more customized architecture is needed. The next chapter zooms out from implementation details entirely, to survey the industries and problem types where MLPs like these actually get deployed.