Context Neural MT Before Attention (2013–2014)

Quantified The Bottleneck Cracks Past 30 Tokens

Insight Let the Decoder Look Back

Instead of compressing into a single fixed bottleneck, compute a step-dependent context for each output position $i$:

Definition The Additive Alignment Score

The Bahdanau Architecture

Encoder Choice Why Bidirectional?

Forward Pass — One Decoder Step Numerically

$T=4$, $d_s=d_h=2$, $d_a=2$. Query $s_{i-1}=(1,0)^\top$. Encoder states and parameters chosen so the math stays small.

Empirical WMT'14 EN→FR — Bahdanau et al. 2014

Soft Alignment EN → FR — the Famous Heatmap

Caveat Is Attention an Explanation?

Limits Three Problems with Additive Attention

• Parameter cost grows with $d_a$. Each score needs an MLP with $d_a (d_s + d_h) + d_a$ parameters. To get expressive scoring you must enlarge $d_a$, and that scales with both encoder and decoder dimensions.
• Cannot batch as a single matmul. The tanh inside $e_{ij} = v_a^\top \tanh(W_a [s_{i-1}; h_j])$ forces a broadcast: tile $s_{i-1}$ across $T$ encoder positions, run the tanh, then reduce. GPUs hate this — it is several kernel launches and a non-fused activation between them.
• Cost per decoder step is $O(T \cdot d_a \cdot (d_s + d_h))$. For autoregressive decoding of length $L$, total cost is $O(L \cdot T \cdot d_a \cdot (d_s + d_h))$ — quadratic in the sequence dimensions and tied to the MLP width.

Legacy What Bahdanau Gave Us

• Soft, differentiable alignment. A learned probability distribution over input positions, trained end-to-end with the rest of the network — the IBM-models dream made gradient-friendly.
• Interpretable attention maps. $\alpha_{ij}$ as a 2D heatmap turned the encoder-decoder from a black box into something a linguist could read — even with the Jain & Wallace caveats.
• A primitive that survives intact. Strip the tanh, swap concatenation for dot-product, add a $1/\sqrt{d_k}$ scaling, share queries across the same sequence — and you have self-attention. Same softmax, same weighted sum.

Next: Ch 38 simplifies the score function (multiplicative, dot-product, scaled dot-product) so attention becomes a single matrix multiplication — the ingredient that lets the Transformer parallelise across an entire sequence.

Luong 2015 One Year After Bahdanau

Definition Four Scoring Functions

• "Concat" in Luong's notation $\equiv$ Bahdanau's additive — same formula, different name
• "General" relaxes the $d_s = d_h$ constraint of plain dot product via a learned bilinear form
• $s = $ decoder query at step $i$, $h = $ encoder hidden at step $j$

Hardware Why Dot Product Wins

Theorem Score Variance Grows with Dimension

Then for each term:

$\mathbb{E}[s_k h_k] = \mathbb{E}[s_k]\mathbb{E}[h_k] = 0,\quad \mathrm{Var}(s_k h_k) = \mathbb{E}[s_k^2]\mathbb{E}[h_k^2] - 0 = 1$

The $d_k$ terms of $s^\top h = \sum_k s_k h_k$ are pairwise independent, so variance adds:

Score magnitude grows as $\sqrt{d_k}$. Dot-product attention is statistically dimension-dependent.

Failure mode Softmax Saturates

Vaswani 2017 Scale by $\sqrt{d_k}$

Dividing by $\sqrt{d_k}$ restores $\mathrm{Var}(e_{ij}) = 1$ regardless of dimension.

• One line of code: scores = Q @ K.T / math.sqrt(d_k)
• Vaswani et al.'s ablation: without scaling, training stalls at $d_k > 64$
• With scaling: comfortable training to $d_k \in \{256, 512, 1024, \ldots\}$
• No new parameters, no new gradient path — the cheapest fix in deep learning

Entropy of Attention vs $d_k$

Reframe Multiplicative = Projected Dot Product

Rewrite Luong's general (multiplicative) score:

• Multiplicative attention is dot-product attention applied to a learned projection of one of its arguments
• Equivalently: project $s$ through $W_g^\top$, then take a plain dot product with $h$
• Hardware cost: one extra matmul, no other architectural change

Luong 2015 Global vs Local Attention

Coverage Stop Re-Attending to the Same Words

Vanilla attention has no memory of where it has already looked $\Rightarrow$ summarisation models repeat phrases.

• Add $\mathrm{cov}_j^{(i)}$ as an extra feature into the attention score
• The $\min$ saturates contribution once a position has been fully consumed
• Critical for abstractive summarisation; less so for word-by-word NMT

WMT EN-DE — Score Function Bake-off

• Numbers approximated from Luong et al. 2015 and Vaswani et al. 2017 ablations — exact comparisons are confounded by tokeniser, batch size, and warm-up
• The scaled-dot-product entry is the only one that combines the speed of GEMM with the stability of additive

Bridge to Ch 39 The Door to Self-Attention

• Parameter-free score function — no MLP weights to tune
• Batchable as a single matmul $S K^\top$ — one GEMM call
• Stable across dimensions thanks to $\sqrt{d_k}$ — trains to $d_k = 1024+$
• Modular — works with any (query, key, value) triple, not just decoder $\times$ encoder

Ch 39: self-attention · Ch 40: the full Transformer

The Leap From Cross- to Self-Attention

Until now, attention has always been between two sequences:

• Bahdanau (Ch 37): the decoder queries the encoder
• Luong (Ch 38): same shape, different score function
• Always: $Q$ comes from one place, $K, V$ from another

This is the boldest move of the course. We are about to remove recurrence entirely — no $h_t = f(h_{t-1}, x_t)$, no BPTT, no hidden state passed through time. A sequence becomes a set of vectors that mix with each other in one parallel matmul.

Definition Q/K/V as a Soft Dictionary

Three Hats from One Vector

Given an input $X \in \mathbb{R}^{T \times d_\text{model}}$ (a sequence of $T$ token embeddings), define three learned linear projections:

• Same input vector $x_i$ produces three different views: $q_i, k_i, v_i$
• $q_i$ — "what am I looking for?"
• $k_i$ — "what do I respond to?"
• $v_i$ — "what do I contribute if attended to?"

Matrix Form The Single Most Important Formula in Modern AI

Stack $T$ queries as the rows of $Q$, similarly for $K, V$:

• $A_{ij}$ = "how much does token $i$ attend to token $j$?"
• Row $i$ of $\mathrm{Out}$ = weighted sum of all values, weighted by row $i$ of $A$
• Two matmuls + one softmax. The whole thing is one fused GPU kernel.

Theorem Self-Attention is Permutation-Equivariant

Proof sketch (two steps):

• (i) $Q' = X' W_Q = P X W_Q = P Q$. Same for $K' = PK$, $V' = PV$.
• (ii) $Q'(K')^\top = (PQ)(PK)^\top = P\,QK^\top P^\top$. Row-wise softmax commutes with row permutation, so $A' = P A P^\top$. Multiply: $A' V' = P A P^\top P V = P A V$. $\;\square$

Per-Layer Complexity: RNN vs Self-Attention vs CNN

• Sequential ops $O(1)$: the whole $A V$ product is one parallel matmul
• Max path length $1$ is the killer feature. Gradient between any two positions traverses exactly one layer — no BPTT chain (Ch 33), no vanishing through time
• $T^2$ beats $T \cdot d^2$ as long as $T < d$ — true for most sentences ($d_\text{model} = 512, T \approx 100$)

Source: Vaswani et al. (2017), Attention Is All You Need, NeurIPS 2017 (arXiv:1706.03762), Table 1.

Cost The $O(T^2)$ Memory Wall

The attention matrix $A \in \mathbb{R}^{T \times T}$ must be materialised for the backward pass.

Historical Detour Schmidhuber 1991 — Fast Weight Programmers

• This is the same Q/K/V structure used in Transformers — twenty-six years earlier
• "Programmer" because the slow net writes the weights of the fast net on the fly
• Trained end-to-end with backprop; just never scaled

Schmidhuber, J. (1992). Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation 4(1), 131-139. (preprint TR FKI-147-91, March 1991, TUM.)

Priority Vaswani 2017 ↔ Schmidhuber 1991

Vaswani et al. (2017) did not cite Schmidhuber 1991. Twenty years later, Schlag, Irie & Schmidhuber proved formal equivalence:

Schlag, I., Irie, K., Schmidhuber, J. (2021). Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021 (arXiv:2102.11174).

Definition Causal (Masked) Self-Attention

Three Uses of Attention in a Transformer

Beautiful unification: one operation, three roles, decided entirely by where $Q, K, V$ come from and what mask is applied.

Definition Multi-Head Attention

A single attention captures one relation. Language has many simultaneously: subject-verb agreement, anaphora, syntactic head, semantic similarity, positional adjacency.

• Each head operates in its own $d_k$-dim subspace
• Total parameter count $\approx$ one big head — no extra cost
• Heads run in parallel; concatenated output goes through $W_O$
• Original paper: $d_\text{model}=512$, $h=8$, $d_k=64$

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., Polosukhin, I. (2017). Attention Is All You Need. NeurIPS 2017 (arXiv:1706.03762), §3.2.2.

What Do Different Heads Actually Learn?

Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019 (arXiv:1905.09418).
Clark, K., Khandelwal, U., Levy, O., Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT's Attention. BlackboxNLP 2019 (arXiv:1906.04341) — heads for direct objects, possessives, coreference, "next-token".

Closing the Loop Self-Attention = Modern Hopfield Update

Ramsauer et al. (2020) proved that one self-attention step is exactly one update of a continuous Modern Hopfield Network with exponential interaction energy:

• Self-attention = one-step associative-memory retrieval
• Exponential capacity in $d$: the modern Hopfield net stores $\sim e^{d/2}$ patterns vs $0.14d$ for the 1982 binary version
• Closes the loop with Ch 32: Hopfield (1982) opened the recurrent series; Hopfield (2020) reappears as the energy view of the Transformer

Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., Gruber, L., Holzleitner, M., Pavlović, M., Sandve, G. K., Greiff, V., Kreil, D., Kopp, M., Klambauer, G., Brandstetter, J., Hochreiter, S. (2020). Hopfield Networks Is All You Need. ICLR 2021 (arXiv:2008.02217).

Vaswani 2017 The Full Encoder–Decoder Architecture

Definition One Encoder Block

• Every sublayer is wrapped in residual + LayerNorm (post-LN convention)
• $\mathrm{MHA}(x, x, x)$ — queries, keys, values are all projections of the same $x$
• $\mathrm{FFN}$ acts independently at each token position (parameter-shared MLP)
• Same shape in, same shape out — blocks compose trivially. Stack $N=6$.

Bidirectional: every position attends to every other position, including the future. Acceptable on the encoder side because the entire input is available at once.

Definition One Decoder Block — Three Sublayers

• Causal mask: add $-\infty$ to the upper triangle of $QK^\top/\sqrt{d_k}$ — future tokens get softmax weight zero
• Cross-attention is the modern descendant of Bahdanau (Ch 37): the decoder asks questions of the encoder

Definition Sinusoidal Positional Encoding

• $p$ — token position; $k = 0, 1, \ldots, d_{\mathrm{model}}/2 - 1$ — frequency index
• Added (not concatenated) to the input embedding before the first block
• Wavelengths form a geometric progression: $\lambda_k = 2\pi \cdot 10000^{2k/d_{\mathrm{model}}}$

Range of scales. For $d = 512$: fastest dim has wavelength $\approx 2\pi \approx 6.3$ tokens; slowest has $\approx 2\pi \cdot 10^4 \approx 62{,}832$ tokens. A Fourier basis from word-level to document-level all in one vector.

Theorem Linear Shift Property of Sinusoidal PE

Proof sketch. Each $(\sin, \cos)$ pair at frequency $\omega_k = 10000^{-2k/d}$ obeys

$\begin{pmatrix}\sin\omega_k(p+\Delta)\\ \cos\omega_k(p+\Delta)\end{pmatrix} = \underbrace{\begin{pmatrix}\cos\omega_k\Delta & \sin\omega_k\Delta\\ -\sin\omega_k\Delta & \cos\omega_k\Delta\end{pmatrix}}_{\text{rotation by } \omega_k \Delta}\begin{pmatrix}\sin\omega_k p\\ \cos\omega_k p\end{pmatrix}.$

Stacking the per-frequency $2\times 2$ rotations gives a block-diagonal $R(\Delta)$. $\;\blacksquare$

Learned PE, Sinusoidal PE, and Their Successors

Vaswani's ablation. Learned and sinusoidal match in-distribution — but learned PE collapses beyond training-distribution length, because the embedding for position 8193 was never trained.

Ba, Kiros, Hinton 2016 Layer Normalization

Ba, Kiros, Hinton. Layer Normalization. arXiv:1607.06450 (2016).

• $\mu, \sigma$ computed across the feature dimension only — one mean and std per token, per sample
• $\gamma, \beta \in \mathbb{R}^{d}$ are learned per-feature scale and shift
• $\epsilon \approx 10^{-5}$ for numerical stability
• No batch dependency — same operation at training and inference, batch size 1 or 1024

This is the only per-layer normalisation in the Transformer. Replaces the BatchNorm of CNNs (Ch 27).

LayerNorm vs BatchNorm — Why Transformers Avoid BN

Residual Connections — the Simplest Gate

$y \;=\; \mathrm{LN}\!\big(x + \mathrm{Sublayer}(x)\big)$

Residual + LayerNorm + warm-up = the deep-stack recipe.

Pre-LN vs Post-LN — the Quiet 2020 Switch

Trade-off. Post-LN squeezes a bit more performance when training succeeds; pre-LN almost always trains. In 2024, "almost always trains" wins.

Definition Position-wise Feed-Forward Network

• Applied identically and independently at each token position — same $W_1, W_2$ shared across all $T$ positions
• $W_1 \in \mathbb{R}^{d_{\mathrm{model}} \times d_{\mathrm{ff}}}$, $W_2 \in \mathbb{R}^{d_{\mathrm{ff}} \times d_{\mathrm{model}}}$
• Original sizes: $d_{\mathrm{model}} = 512$, $d_{\mathrm{ff}} = 2048$ — factor-of-4 expansion
• Holds $\approx 67\%$ of all Transformer parameters — the FFN is where the model "stores" knowledge

Sparse FFN. Replace one big FFN with $N$ smaller "experts" + a router — Switch Transformer (Fedus 2021, JMLR, arXiv:2101.03961), Mixtral 8×7B (2023). Same compute, much more parameters.

Training Recipe — the Three Indispensable Tricks

Empirical Results — Vaswani et al. 2017, Table 2

• WMT 2014 newstest, BLEU on cased detokenised output
• SOTA on both pairs, with $\sim 1/4$ the training compute of the previous best (ConvS2S) on EN–FR
• Training time: 12 hours on 8×P100 for base, 3.5 days for big

Why This Architecture Won — Three Structural Reasons

Decoder-only — the GPT Lineage

Encoder-only — BERT and Friends