ML from Scratch · Lecture 3 of 5

Neural Networks I: From the Perceptron to the Multilayer Network

A neuron is a weighted sum and a squashing function. One of them can only draw a straight line; stack them with a non-linearity in between and they can draw anything.

The neuron

A single unit computes a weighted sum of its inputs plus a bias, then applies an activation function $\phi$:

With $\phi=$ the sigmoid $\sigma(z)=1/(1+e^{-z})$ this is exactly logistic regression: the output is a probability, and the decision boundary $w^\top x+b=0$ is a straight line (a hyperplane).

The perceptron and its limit

Rosenblatt's perceptron (1958) uses a step function and the update $w\leftarrow w+\eta(y-\hat y)x$. It converges whenever the classes are linearly separable. Minsky and Papert (1969) pointed out that it cannot learn XOR: points $(0,0),(1,1)$ in one class and $(0,1),(1,0)$ in the other. No single line separates them. This result cooled neural network research for over a decade.

Hidden layers

Put a layer of $H$ neurons between input and output:

Each hidden unit draws its own line. The output unit combines these half-planes into regions of any shape. Two hidden units suffice for XOR: one detects "x₁ OR x₂", the other "x₁ AND x₂", and the output fires for OR-but-not-AND.

Press ▶ with the XOR data. With 1 hidden unit the network fails (it is still one line). With 2–4 tanh units it carves out the checkerboard. Switch the activation to none (linear) and it fails again however many units you add, which leads to the next point.

Why the non-linearity is essential

Without $\phi$, two layers collapse into one: $W^{(2)}(W^{(1)}x+b^{(1)})+b^{(2)}=Wx+b$. Depth buys nothing. Common activations:

• Sigmoid $\sigma(z)$: smooth, outputs in (0,1); saturates, so gradients vanish for large |z|.
• tanh: zero-centred version, $\tanh z=2\sigma(2z)-1$.
• ReLU $\max(0,z)$: cheap, doesn't saturate for z>0, and is the default for deep nets. Variants: Leaky ReLU, GELU.

Universal approximation

A network with one hidden layer and enough units can approximate any continuous function on a compact set to arbitrary accuracy (Cybenko 1989; Hornik 1991). The intuition: two shifted steep sigmoids make a "bump", and enough bumps tile any function, like a histogram. The theorem says nothing about how many units you need or whether gradient descent will find them. In practice, depth represents many functions exponentially more efficiently than width.

Forward pass in NumPy

def forward(X, W1, b1, w2, b2):
    Z1 = X @ W1.T + b1          # (n, H)
    H  = np.tanh(Z1)
    s  = H @ w2 + b2            # (n,)
    p  = 1 / (1 + np.exp(-s))
    return Z1, H, p

def loss(p, y):                 # binary cross-entropy
    return -np.mean(y * np.log(p) + (1 - y) * np.log(1 - p))

Training means adjusting $W^{(1)},b^{(1)},w^{(2)},b^{(2)}$ to reduce the loss by gradient descent. That needs the gradient of the loss with respect to every weight, and computing it efficiently is backpropagation, the subject of the next lecture.