ML from Scratch · Lecture 4 of 5

Neural Networks II: Backpropagation from Scratch

The chain rule, organised so that one backward sweep gives every gradient. Worked by hand on a tiny network, then vectorised.

November 15, 2026By Diganta Bhattacharya5 min read

The problem

A network is a composition $L=\ell(f_K(\cdots f_2(f_1(x;\theta_1);\theta_2)\cdots))$. Gradient descent needs $\partial L/\partial\theta$ for every parameter. Doing each by finite differences costs one forward pass per parameter, millions of passes. Backpropagation gets all of them for the cost of about two passes.

The chain rule on a computational graph

Write the forward pass as a graph of simple operations. For our network from Lecture 3:

$$z=Wx+b\;\to\;h=\tanh(z)\;\to\;s=u^\top h+c\;\to\;p=\sigma(s)\;\to\;L=-[y\ln p+(1-y)\ln(1-p)].$$

Going backwards, each node multiplies the incoming gradient by its local derivative:

$$\frac{\partial L}{\partial s}=p-y\quad(\text{sigmoid + cross-entropy simplify beautifully})$$ $$\frac{\partial L}{\partial u}=\frac{\partial L}{\partial s}\,h,\qquad \frac{\partial L}{\partial c}=\frac{\partial L}{\partial s},\qquad \frac{\partial L}{\partial h}=\frac{\partial L}{\partial s}\,u$$ $$\frac{\partial L}{\partial z}=\frac{\partial L}{\partial h}\odot(1-h^2),\qquad \frac{\partial L}{\partial W}=\frac{\partial L}{\partial z}\,x^\top,\qquad \frac{\partial L}{\partial b}=\frac{\partial L}{\partial z}.$$

Derivation of ∂L/∂s = p − y$\frac{\partial L}{\partial p}=-\frac yp+\frac{1-y}{1-p}=\frac{p-y}{p(1-p)}$ and $\frac{\partial p}{\partial s}=p(1-p)$. Multiply and the denominator cancels. This is why sigmoid outputs are paired with cross-entropy (and softmax with categorical cross-entropy): the gradient is just "prediction − target".

Worked example

Below is a 2-input, 2-hidden-unit, 1-output network with fixed weights $W=\begin{pmatrix}0.5&-0.3\\0.8&0.2\end{pmatrix}$, $b=(0.1,-0.1)$, $u=(1.2,-0.7)$, $c=0.05$. Move the inputs and target and every forward value and gradient updates. Try checking one entry by hand, e.g. $\partial L/\partial W_{11}=\frac{\partial L}{\partial z_1}x_1$.

Vectorised for a mini-batch

def backward(X, y, Z1, H, p, w2):
    n = len(y)
    ds  = (p - y) / n                    # (n,)
    dw2 = H.T @ ds                       # (H,)
    db2 = ds.sum()
    dH  = np.outer(ds, w2)               # (n, H)
    dZ1 = dH * (1 - H ** 2)              # tanh'
    dW1 = dZ1.T @ X                      # (H, d)
    db1 = dZ1.sum(axis=0)
    return dW1, db1, dw2, db2

for epoch in range(5000):
    Z1, H, p = forward(X, W1, b1, w2, b2)
    dW1, db1, dw2, db2 = backward(X, y, Z1, H, p, w2)
    W1 -= lr * dW1; b1 -= lr * db1; w2 -= lr * dw2; b2 -= lr * db2

This is exactly what runs inside the training widget of Lecture 3.

Always gradient-check

Compare backprop with a centred finite difference on a few random weights:

$$\frac{\partial L}{\partial\theta}\approx\frac{L(\theta+\epsilon)-L(\theta-\epsilon)}{2\epsilon},\qquad \text{relative error}\lesssim10^{-6}\text{ with }\epsilon=10^{-5}.$$

Vanishing and exploding gradients

Backprop multiplies one local derivative per layer. Sigmoid's derivative is at most ¼, so through 20 layers gradients shrink like $4^{-20}$ and early layers stop learning. Large weights do the opposite and gradients explode. Remedies (next lecture): ReLU activations, careful initialisation, normalisation layers, residual connections and gradient clipping.

Automatic differentiationPyTorch and JAX build this graph for you and run reverse-mode autodiff, which is backprop generalised to any program. Writing it by hand once is the best way to stop treating loss.backward() as magic.

Key takeaways

Backprop is the chain rule applied right to left on the computational graph: all gradients in ~2 passes.
Sigmoid/softmax + cross-entropy give the clean output gradient p − y.
Gradient-check with centred differences; watch for vanishing/exploding gradients in deep nets.

Exercises

Extend the backward pass to a second hidden layer and verify it with a gradient check.
Derive the gradient of softmax + categorical cross-entropy and show it equals p − y (one-hot y).
Why does initialising all weights to zero break learning in a hidden layer? (Think about symmetry.)