This chapter reviews the deep learning module: MLPs (4.1), CNNs (4.2), autoencoders (4.3) and transformers (4.4). The skills to practice are a forward pass through an MLP or a small CNN, backprop through a small MLP, counting parameters, and explaining the core ideas behind each architecture. Practice problems with fold-out answers are at the end.
The module revolved around two questions:
The first question has one answer for every architecture: SGD + backprop. The second has no single answer. Each lecture after 4.1 gave one example of an architecture designed for a particular kind of data:
| lecture | architecture | used for |
|---|---|---|
| 4.1 | MLP (1986) | general architecture |
| 4.2 | CNN (1989) | images |
| 4.3 | Autoencoder (2006) | unsupervised learning |
| 4.4 | Transformer (2017) | sequences |
In practice, we use mini-batch SGD, which averages the gradient over a small batch of examples before each update.
Example: backpropagation with squared loss. In the 4.1 example, the network had \(W_1 = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\), \(b_1 = 0\), \(a = (1, 1)\) and \(c = 0\). The training sample was \(x = (1, 0)\), \(y = 1\). The forward pass gives \(h_1 = (1, 1)\) and \(\hat y = 2\). Working backwards:
\[ \frac{\partial l}{\partial \hat y} = 2(\hat y - y) = 2, \qquad \frac{\partial l}{\partial a} = \frac{\partial l}{\partial \hat y} \frac{\partial \hat y}{\partial a} = 2h_1 = \begin{bmatrix} 2 \\ 2 \end{bmatrix}, \qquad \frac{\partial l}{\partial c} = \frac{\partial l}{\partial \hat y} \frac{\partial \hat y}{\partial c} = 2. \]
SGD with learning rate \(\eta = 0.5\) (updating the output layer only): \[ a \leftarrow \begin{bmatrix} 1 \\ 1 \end{bmatrix} - 0.5 \begin{bmatrix} 2 \\ 2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \qquad c \leftarrow 0 - 0.5 \times 2 = -1. \]
For the hidden layer, the steps are in 4.1. Practice problem 2 below works through a case where ReLU blocks part of the gradient.
Make sure you can: multiply a \(2 \times 2\) matrix by a vector, do a forward pass, and apply the chain rule layer by layer. Always check the shapes: a gradient has the same shape as its parameter.
A CNN is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.
Building blocks:
Example: forward pass. For input \(\begin{bmatrix} 1 & 0 & 2 \\ 0 & 1 & 1 \\ 2 & 1 & 0 \end{bmatrix}\), filter \(K = \begin{bmatrix} 1 & -2 \\ 0 & 1 \end{bmatrix}\) with ReLU, flattening by rows, and a linear layer with \(w = (1, -1, 2, 1)\) and \(b = -1\): \[ \begin{bmatrix} 2 & 0 \\ 0 & 0 \end{bmatrix} \xrightarrow{\text{flatten}} (2, 0, 0, 0) \xrightarrow{\text{linear}} 2 - 1 = 1. \]
Case study: LeNet-5.
| layer | learnable parameters |
|---|---|
| convolution: 6 filters of size \(5 \times 5\) | \(6 \times (5 \times 5) = 150\) |
| pooling | N/A |
| convolution: 16 filters of size \(6 \times (5 \times 5)\) | \(16 \times (6 \times 5 \times 5) = 2{,}400\) |
| pooling | N/A |
| MLP: \(\mathbb{R}^{400} \to \mathbb{R}^{120} \to \mathbb{R}^{84} \to \mathbb{R}^{10}\) | \((120 \times 400 + 120) + (84 \times 120 + 84) + (10 \times 84 + 10) \approx 59{,}000\) |
Total: about 60,000 parameters, learned using SGD + backprop from data.
Why convolution? It respects the 2D structure of the image, and it reuses a few weights at every position. With fewer learnable parameters, a network can reach similar performance with less data. In practice, convolutions usually use padding (which is why LeNet’s first convolution keeps the \(28 \times 28\) size) and convolutions and pooling often use a stride. You don’t need to know the details.
Setup. \[ x \in \mathbb{R}^d \;\xrightarrow{\ \phi\ }\; \phi(x) \in \mathbb{R}^k \;\xrightarrow{\ \phi'\ }\; \phi'(\phi(x)) \in \mathbb{R}^d \]
An autoencoder is a nonlinear generalization of PCA. PCA’s encoder \(U\) and decoder \(U^\top\) become two neural networks, and we lose the closed-form solution but gain flexibility.
Case study: Autoencoder (2006).
The variants extend the idea. A denoising autoencoder reconstructs clean inputs from corrupted ones. A variational autoencoder shapes the latent space so that decoding samples from \(N(0, I_k)\) generates realistic new data.
You won’t be asked to write code, but you should be familiar with the
pipeline from the notebook: define the network, choose a loss and an
optimizer, and loop over mini-batches doing a forward pass,
loss.backward() and an optimizer step.
Toy example: a character-level language model.
The three core ideas:
Parameter count for the toy model:
The transformer is a complicated model, and our toy example covers only its core ideas. Focus on the ideas we discussed explicitly: what each component does, and why it’s needed.
Problem 2 is the bonus example from the recap lecture. The other problems were written for this book; they are not from past exams.
1. MLP forward pass. An MLP with one hidden layer and ReLU activation has \(W_1 = \begin{bmatrix} 1 & -1 \\ -2 & 1 \end{bmatrix}\), \(b_1 = \begin{bmatrix} 0 \\ -1 \end{bmatrix}\), \(a = \begin{bmatrix} 1 \\ -2 \end{bmatrix}\) and \(c = 0.5\). What does it output for \(x = \begin{bmatrix} 2 \\ 1 \end{bmatrix}\)?
\(W_1 x + b_1 = (2 - 1 + 0, \; -4 + 1 - 1) = (1, -4)\). ReLU zeroes out the negative entry, so \(h_1 = (1, 0)\). The output is \(a \cdot h_1 + c = 1(1) + (-2)(0) + 0.5 = 1.5\).
2. Backprop when ReLU is off. Use the network \(W = \begin{bmatrix} 1 & 1 \\ 1 & 1 \end{bmatrix}\), \(b = \begin{bmatrix} -2 \\ 0 \end{bmatrix}\), \(a = \begin{bmatrix} 1 \\ 1 \end{bmatrix}\), \(c = 0\), with \(z = Wx + b\), \(h = \sigma_{\text{ReLU}}(z)\) and \(\hat y = a \cdot h + c\). Take the training sample \(x = \begin{bmatrix} 1 \\ 0 \end{bmatrix}\), \(y = 0\) and squared loss. Compute the gradients with respect to \(a\), \(c\), \(W\) and \(b\).
Forward pass. \(z = (1 - 2, \; 1 + 0) = (-1, 1)\), so \(h = (0, 1)\) and \(\hat y = 1\).
Output layer. \(\dfrac{\partial l}{\partial \hat y} = 2(\hat y - y) = 2\). Then \(\dfrac{\partial l}{\partial a} = 2h = \begin{bmatrix} 0 \\ 2 \end{bmatrix}\) and \(\dfrac{\partial l}{\partial c} = 2\).
Hidden layer. \(\dfrac{\partial l}{\partial h} = 2a = \begin{bmatrix} 2 \\ 2 \end{bmatrix}\). ReLU’s derivative is 0 where \(z < 0\) and 1 where \(z > 0\). Multiplying element-wise (\(\odot\)): \[ \frac{\partial l}{\partial z} = \frac{\partial l}{\partial h} \odot \sigma'_{\text{ReLU}}(z) = \begin{bmatrix} 2 \\ 2 \end{bmatrix} \odot \begin{bmatrix} 0 \\ 1 \end{bmatrix} = \begin{bmatrix} 0 \\ 2 \end{bmatrix}. \] So \[ \frac{\partial l}{\partial W} = \frac{\partial l}{\partial z} \, x^\top = \begin{bmatrix} 0 \\ 2 \end{bmatrix} \begin{bmatrix} 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & 0 \\ 2 & 0 \end{bmatrix}, \qquad \frac{\partial l}{\partial b} = \begin{bmatrix} 0 \\ 2 \end{bmatrix}. \] The first hidden unit is “off” for this input, so its incoming weights and bias get no gradient from this example.
3. Too big a step? In the 4.1 example, SGD with \(\eta = 0.5\) on the output layer gave \(a = (0, 0)\) and \(c = -1\). What does the updated network predict for \(x = (1, 0)\), and what is the new loss? What does this suggest about the learning rate?
The hidden layer didn’t change, so \(h_1 = (1, 1)\) still, and \(\hat y = 0 + 0 - 1 = -1\). The new loss is \((-1 - 1)^2 = 4\), worse than the original loss of \((2 - 1)^2 = 1\). The step overshot: the prediction went from 1 too high to 2 too low. The gradient gives the right direction, but \(\eta = 0.5\) is too large here. In 4.1, \(\eta = 0.1\) with all four parameters updated moved the prediction from 2 to 0.76.
4. Convolution forward pass. Apply the filter \(\begin{bmatrix} 1 & 0 \\ -1 & 1 \end{bmatrix}\) (no padding, stride 1) followed by ReLU to \(\begin{bmatrix} 1 & 2 & 0 \\ 0 & 1 & 3 \\ 2 & 0 & 1 \end{bmatrix}\). Then flatten by rows and apply a linear layer with \(w = (1, 0, -1, 1)\) and \(b = 0\).
After ReLU, the feature map is \(\begin{bmatrix} 2 & 4 \\ 0 & 2 \end{bmatrix}\). Flattening by rows gives \((2, 4, 0, 2)\), and the output is \(2 + 0 - 0 + 2 = 4\).
5. Counting parameters in a CNN. A color image of size \(3 \times 32 \times 32\) (RGB) goes through a convolution layer with 8 filters of size \(3 \times 3\) (no padding, stride 1), then \(2 \times 2\) max pooling with stride 2. (a) How many learnable parameters does the convolution have, ignoring biases? (b) What is the shape of the output after pooling? (c) How many weights would a dense layer need to map the flattened image to a vector of the same size as the convolution’s output?
The input has 3 channels, so each filter is \(3 \times 3 \times 3\), and there are 8 filters: \(8 \times 27 = 216\) parameters (224 with one bias per filter).
A \(3 \times 3\) filter without padding shrinks \(32 \times 32\) to \(30 \times 30\), so the convolution outputs \(8 \times 30 \times 30\). Pooling halves each spatial dimension: \(8 \times 15 \times 15\).
The flattened image has \(3{,}072\) entries and the convolution’s output has \(7{,}200\), so a dense layer would need \(7{,}200 \times 3{,}072 \approx 22\) million weights, compared with 216.
6. Autoencoders and PCA. (a) If an autoencoder’s encoder and decoder are each a single linear layer with no nonlinearity, how does it relate to PCA? (b) Why do modern autoencoders not tie the decoder weights to the transposes of the encoder weights?
The reconstruction is then a linear map of rank at most \(k\), and minimizing squared reconstruction error over such maps finds the same \(k\)-dimensional subspace as PCA. Without PCA’s orthonormality constraint, the learned encoder is just some basis for that subspace, not necessarily the principal directions. The nonlinearities are what let an autoencoder do more than PCA.
In PCA, \(U^\top\) undoes \(U\) because \(U\) has orthonormal rows. A deep encoder composes unconstrained matrices with nonlinearities, so the transposes of its weights have no reason to invert it. It works better to make the decoder a separate network and learn it from data.
7. Attention by hand. Two tokens have embeddings \((1, 0)\) and \((0, 2)\). With \(W_Q = W_K = W_V = I\) (so \(d_k = 2\)), compute \(\text{Attention}(E)\). Use \(e^{0.71} \approx 2.03\) and \(e^{2.83} \approx 16.9\).
\(Q = K = V = E\), so \[ \frac{QK^\top}{\sqrt 2} = \frac{1}{\sqrt 2} \begin{bmatrix} 1 & 0 \\ 0 & 4 \end{bmatrix} \approx \begin{bmatrix} 0.71 & 0 \\ 0 & 2.83 \end{bmatrix}. \] The row-wise softmax gives:
So \[ A \approx \begin{bmatrix} 0.67(1, 0) + 0.33(0, 2) \\ 0.06(1, 0) + 0.94(0, 2) \end{bmatrix} = \begin{bmatrix} 0.67 & 0.66 \\ 0.06 & 1.89 \end{bmatrix}. \] The second token’s key has a large dot product with its own query, so it mostly keeps its own information. The first token borrows a third of its new embedding from the second token.
8. Why positional encoding? Explain why a model built from input embeddings and attention alone could not tell “cat” from “act” when predicting the next character.
Attention is permutation-equivariant. Reordering the input rows reorders the output rows in the same way, and nothing else changes. “cat” and “act” contain the same three tokens, so the updated embeddings would be the same three vectors, just in a different order. In particular, both words end in “t”, and t’s updated embedding is a weighted average over the same three tokens either way, so it comes out identical. Since the prediction uses only the last token’s embedding, the model would predict exactly the same next-character distribution for both. Adding a learned position vector to each embedding fixes this.
9. Counting transformer parameters. For the toy language model with context length \(T = 32\), \(d = d' = 16\) and \(d_k = 8\), how many parameters are there?
The total is \(1{,}915\). If we tokenized by words, the input embedding (and the output layer) would grow with the vocabulary size, which can be tens of thousands of tokens.