Part V
Backpropagation
The Algorithm That Changed Everything
Chapters 15–19
Part V covers the mathematical engine behind modern neural networks. We start with gradient descent as an optimization framework, derive the backpropagation equations from the chain rule, analyze activation functions and the vanishing gradient problem, discuss practical considerations, and conclude with the Universal Approximation Theorem.
Definition Empirical Risk Minimization
ERM objective: find parameters that minimize average loss over training data:
\[\min_{\mathbf{w}} \; \mathcal{L}(\mathbf{w}) = \frac{1}{N} \sum_{i=1}^{N} L\bigl(f(\mathbf{x}^{(i)};\mathbf{w}),\, \mathbf{y}^{(i)}\bigr)\]
\(\mathbf{w}\): learnable parameters (weights + biases)
\(L\): per-sample loss (measures prediction error)
Learning = finding \(\mathbf{w}^*\) that minimizes average loss
Chapter 15
Full Notes →
The ERM framework is the bridge between machine learning and optimization. Instead of hand-crafting rules, we define a loss function and let an algorithm find the best parameters. The loss L measures how far the prediction f(x; w) is from the true label y. We average over all N training samples to get the empirical risk. This reformulates LEARNING as OPTIMIZATION.
Algorithm Gradient Descent
\[\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \,\nabla\mathcal{L}(\mathbf{w}_t)\]
Gradient \(\nabla\mathcal{L}\) points in direction of steepest ascent
Negative gradient \(-\nabla\mathcal{L}\) gives steepest descent direction
Chapter 15
Full Notes →
This is the core update rule of gradient descent. At each step t, we compute the gradient of the loss at the current parameters, multiply by the learning rate eta, and subtract from the current parameters. The gradient points uphill, so the negative gradient points downhill — toward lower loss. The learning rate eta controls step size.
Geometry Gradient & Contour Lines
\(w_1\)
\(w_2\)
min
start
\(\nabla\mathcal{L}\)
\(-\nabla\mathcal{L}\)
contour lines: constant \(\mathcal{L}\)
GD path (zigzag toward minimum)
Chapter 15
Full Notes →
This contour plot shows a 2D loss surface. Each ellipse represents a set of parameter values with the same loss. The gradient at any point is perpendicular to the contour line and points toward higher loss (red arrow). The negative gradient points toward lower loss (green arrow). The GD path zigzags because the gradient direction changes at each step. Note how the path oscillates across the narrow valley — this motivates momentum-based methods.
Hyperparameter Learning Rate Effects
Too small
...
min
Slow convergence
Just right
min
Smooth convergence
Too large
min
Divergence
Chapter 15
Full Notes →
The learning rate eta is the most critical hyperparameter. Too small: convergence is painfully slow, many steps barely move toward the minimum. Just right: smooth convergence in a reasonable number of steps. Too large: the algorithm overshoots the minimum, oscillates wildly, and can diverge to infinity. Finding the right learning rate is often the first tuning step in practice.
Landscape Convex vs Non-Convex
Convex
global min
Unique global minimum
GD converges for proper \(\eta\)
Non-Convex
local
global
saddle
local
Multiple local minima
Saddle points
Neural networks are non-convex
Chapter 15
Full Notes →
Convex loss surfaces have a single global minimum — GD is guaranteed to find it with a proper learning rate. Non-convex surfaces have multiple local minima and saddle points. Neural network loss landscapes are highly non-convex, so GD may converge to local minima or get stuck at saddle points. In practice, SGD with momentum often works well despite non-convexity.
Definition Loss Functions
Mean Squared Error
Cross-Entropy
Both gradients have the elegant form: prediction − target . Error drives learning.
Chapter 15
Full Notes →
The two most common loss functions. MSE is used for regression; the 1/2 factor simplifies the derivative. Cross-entropy is used for classification, especially with softmax output. Both produce gradients that are just prediction minus target — a beautifully simple and interpretable error signal that drives learning.
Comparison Batch vs SGD vs Mini-batch
Method
Cost/iteration
Convergence
Notes
Batch GD \(O(N)\)
\(O(1/t)\)
Smooth, exact gradient
SGD \(O(1)\)
\(O(1/\sqrt{t})\)
Fast updates, noisy
Mini-batch \(O(B)\)
Between
Practical standard
Mini-batch (B = 32–256) balances computational efficiency and gradient quality.
Chapter 15
Full Notes →
Batch GD uses all N samples per update — exact gradient but expensive. SGD uses a single sample — very fast but noisy. Mini-batch SGD is the practical standard, using B samples per update (typically 32-256). The noise in SGD/mini-batch can actually help escape local minima. Modern training uses mini-batch with adaptive learning rates (Adam).
From Optimization to Networks
Now we have gradient descent — but how do we compute gradients through a multi-layer network?
We've established the optimization framework: minimize a loss function using gradient descent. But for neural networks with multiple layers, computing the gradient is not trivial. We need the chain rule applied systematically — this is backpropagation.
Notation Network Architecture
\(a_1^{(0)}\)
\(a_2^{(0)}\)
\(a_3^{(0)}\)
Input
\(\mathbf{a}^{(0)}\)
\(a_1^{(1)}\)
\(a_2^{(1)}\)
\(a_3^{(1)}\)
\(a_4^{(1)}\)
Hidden
\(a_1^{(2)}\)
\(a_2^{(2)}\)
Output
\(\mathbf{a}^{(2)}\)
\(\mathbf{W}^{(1)}\)
\(\mathbf{W}^{(2)}\)
\(z_1^{(1)}\)
\(z_2^{(1)}\)
\(z_1^{(2)}\)
\(z_2^{(2)}\)
Chapter 16
Full Notes →
The standard notation for a multi-layer network. Superscript (l) denotes the layer. a^(0) is the input, z^(l) is the pre-activation (linear combination), a^(l) is the post-activation. Weight matrix W^(l) connects layer l-1 to layer l. This notation makes the backpropagation equations clean and general.
Algorithm Forward Pass
\[\mathbf{z}^{(l)} = \mathbf{W}^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}\]
\[\mathbf{a}^{(l)} = \sigma^{(l)}\bigl(\mathbf{z}^{(l)}\bigr)\]
Pre-activation \(\mathbf{z}\): linear transformation (matrix multiply + bias)
Post-activation \(\mathbf{a}\): nonlinear squashing (sigmoid, ReLU, etc.)
Chapter 16
Full Notes →
The forward pass computes the output layer by layer, from input to output. At each layer l, we first compute the pre-activation z as a linear combination of the previous layer's activations, then apply the nonlinear activation function sigma. This separation into linear and nonlinear steps is key to the clean derivation of backpropagation. All intermediate values z and a are cached for the backward pass.
Theorem Chain Rule
The multivariate chain rule for composed functions:
\[\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \frac{\partial \mathcal{L}}{\partial z_i^{(l)}} \cdot \frac{\partial z_i^{(l)}}{\partial w_{ij}^{(l)}} = \delta_i^{(l)} \cdot a_j^{(l-1)}\]
Key insight: define the error signal \(\delta_i^{(l)} = \partial\mathcal{L}/\partial z_i^{(l)}\) — the sensitivity of the loss to each pre-activation.
Once we know all \(\delta\) values, weight gradients follow immediately
The challenge: computing \(\delta\) efficiently layer by layer
Chapter 16
Full Notes →
The chain rule is the mathematical foundation of backpropagation. We decompose the partial derivative of the loss with respect to each weight into two factors: the error signal delta (how sensitive the loss is to the pre-activation) and the input activation (from the previous layer). The delta values are the key quantities — once we have them, all gradients follow by simple multiplication.
BP1 Output Layer Error
\[\boxed{\boldsymbol{\delta}^{(L)} = \nabla_{\mathbf{a}^{(L)}}\mathcal{L} \odot \sigma'^{(L)}\bigl(\mathbf{z}^{(L)}\bigr)}\]
First factor: how loss changes with output activation
Second factor: how activation changes with pre-activation (\(\odot\) = element-wise product)
This is the starting point of the backward pass — error begins at the output.
Chapter 16
Full Notes →
BP1 is the first backpropagation equation. It computes the error signal at the output layer L. The first factor is the derivative of the loss with respect to the output activations — for MSE this is (y_hat - y). The second factor is the derivative of the activation function evaluated at the pre-activation z. The Hadamard product combines them element-wise. This is where error information originates.
BP2 Error Backpropagation
\[\boxed{\boldsymbol{\delta}^{(l)} = \bigl((\mathbf{W}^{(l+1)})^\top \boldsymbol{\delta}^{(l+1)}\bigr) \odot \sigma'^{(l)}\bigl(\mathbf{z}^{(l)}\bigr)}\]
Error propagates backward through transpose weights
Each layer's error depends on the next layer's error (recursive)
The transpose \((\mathbf{W}^{(l+1)})^\top\) distributes error back to contributing neurons.
Chapter 16
Full Notes →
BP2 is the recursive step — the heart of backpropagation. It expresses the error at layer l in terms of the error at layer l+1. The transpose weight matrix distributes error from the next layer back to the current layer's neurons, proportional to how much each neuron contributed. The sigma' factor accounts for how much the activation function attenuated the signal. This recursion runs backward from the output to the first hidden layer.
BP3 + BP4 Weight & Bias Gradients
\[\boxed{\frac{\partial\mathcal{L}}{\partial\mathbf{W}^{(l)}} = \boldsymbol{\delta}^{(l)}\bigl(\mathbf{a}^{(l-1)}\bigr)^\top}\]
\[\boxed{\frac{\partial\mathcal{L}}{\partial\mathbf{b}^{(l)}} = \boldsymbol{\delta}^{(l)}}\]
BP3: weight gradient = error \(\times\) input (outer product)BP4: bias gradient = error signal directly
Chapter 16
Full Notes →
BP3 and BP4 convert error signals into parameter gradients. BP3 says the weight gradient is the outer product of the error signal delta and the input activations from the previous layer. This is a Hebbian-like rule: weights change proportionally to how much the input contributed and how wrong the output was. BP4 is even simpler: the bias gradient equals the error signal directly, since the bias always has input 1.
Reference The Four BP Equations
#
Name
Equation
Purpose
BP1 Output Error
\(\boldsymbol{\delta}^{(L)} = \nabla_{\mathbf{a}}\mathcal{L} \odot \sigma'(\mathbf{z}^{(L)})\)
Start the backward pass
BP2 Error Backprop
\(\boldsymbol{\delta}^{(l)} = (W^{(l+1)})^\top\boldsymbol{\delta}^{(l+1)} \odot \sigma'(\mathbf{z}^{(l)})\)
Propagate to earlier layers
BP3 Weight Gradient
\(\partial\mathcal{L}/\partial W^{(l)} = \boldsymbol{\delta}^{(l)}(\mathbf{a}^{(l-1)})^\top\)
How to update weights
BP4 Bias Gradient
\(\partial\mathcal{L}/\partial \mathbf{b}^{(l)} = \boldsymbol{\delta}^{(l)}\)
How to update biases
These four equations are all you need. Everything else is bookkeeping.
Chapter 16
Full Notes →
This is the complete reference for the four backpropagation equations. BP1 initializes the error at the output. BP2 propagates it backward layer by layer. BP3 and BP4 convert errors into parameter gradients. Together, they allow us to compute the gradient of the loss with respect to every parameter in the network in a single backward pass. This is the algorithm that made deep learning practical.
Algorithm Backpropagation PseudocodeBACKPROPAGATION(X, Y, network):
// Forward pass
for l = 1 to L:
z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾
a⁽ˡ⁾ = σ⁽ˡ⁾(z⁽ˡ⁾)
// Backward pass
δ⁽ᴸ⁾ = ∇ₐL ⊙ σ'(z⁽ᴸ⁾) // BP1
for l = L-1 down to 1:
δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾)ᵀδ⁽ˡ⁺¹⁾ ⊙ σ'(z⁽ˡ⁾) // BP2
// Update
for l = 1 to L:
W⁽ˡ⁾ -= η · δ⁽ˡ⁾(a⁽ˡ⁻¹⁾)ᵀ // BP3
b⁽ˡ⁾ -= η · δ⁽ˡ⁾ // BP4
Chapter 16
Full Notes →
The complete backpropagation algorithm in pseudocode. Three phases: (1) Forward pass — compute all z and a values layer by layer. (2) Backward pass — compute error signals delta from output to input using BP1 and BP2. (3) Update — use BP3 and BP4 to adjust weights and biases. This entire procedure is repeated for each mini-batch until convergence.
Insight Computational Complexity
Backpropagation cost:
Forward pass: \(O(L \cdot n^2)\) for \(L\) layers of width \(n\)
Backward pass: same complexity (~2× forward cost)
Alternative: finite differences would need \(O(P)\) forward passes (one per parameter).~500,000× faster .
Backprop trades memory (storing all activations) for speed (one backward pass).
Chapter 16
Full Notes →
The efficiency of backpropagation is what makes neural networks trainable. Computing gradients via finite differences would require 2P forward passes (two per parameter), which is prohibitively expensive for networks with millions of parameters. Backpropagation computes ALL gradients in a single backward pass, at roughly twice the cost of a single forward pass. The tradeoff is memory: we must store all intermediate activations for the backward pass.
History Backpropagation Timeline
Year
Contributors
Contribution
1970 Linnainmaa
Reverse-mode automatic differentiation
1974 Werbos
Applied to neural networks (PhD thesis)
1982 Hopfield
Revived neural networks via physics
1986 Rumelhart, Hinton & Williams
Nature paper popularizing backprop
The algorithm was discovered multiple times independently — but the 1986 paper gave the community a practical tool and clear exposition.
Chapter 16
Full Notes →
Backpropagation has a rich history of independent rediscovery. Linnainmaa described the mathematical core (reverse-mode AD) in 1970. Werbos applied it to neural networks in 1974, but his work wasn't widely known. After Hopfield revived interest in neural networks, Rumelhart, Hinton, and Williams published their seminal 1986 Nature paper that made backpropagation the standard training algorithm.
Definition Sigmoid Activation
\[\sigma(x) = \frac{1}{1+e^{-x}}\]
\[\sigma'(x) = \sigma(x)(1-\sigma(x))\]
Range: \((0,1)\)
Max derivative = \(1/4\) at \(x=0\)
Saturates at extremes
Chapter 17
Full Notes →
The sigmoid function was the standard activation in early neural networks. It maps any real number to (0,1), making it interpretable as a probability. Its derivative has the elegant form sigma(x)(1-sigma(x)), which is maximal at x=0 (value 1/4) and approaches 0 at the extremes. This saturation is a key weakness: for large |x|, the gradient nearly vanishes.
Definition Tanh Activation
\[\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}\]
\[\tanh'(x) = 1 - \tanh^2(x)\]
Range: \((-1,1)\)
Zero-centered (unlike sigmoid)
Max derivative = 1 at \(x=0\)
Chapter 17
Full Notes →
Tanh is a rescaled sigmoid: tanh(x) = 2*sigma(2x) - 1. Its key advantage is being zero-centered, which helps gradient descent converge faster because the outputs are balanced around 0. The maximum derivative is 1 (vs 1/4 for sigmoid), which reduces gradient attenuation. However, tanh still saturates at extremes, so it still suffers from the vanishing gradient problem for deep networks.
Definition ReLU Activation
x
y
kink at 0
gradient = 0
gradient = 1
\[\text{ReLU}(x) = \max(0,x)\]
\[\text{ReLU}'(x) = \begin{cases}1 & x > 0 \\ 0 & x \leq 0\end{cases}\]
Non-saturating (gradient = 1 for \(x>0\))
Computationally cheap
Problem: dying ReLU (neurons stuck at 0)
Chapter 17
Full Notes →
ReLU (Rectified Linear Unit) was the breakthrough activation function. For positive inputs, the gradient is exactly 1, solving the vanishing gradient problem. It's also computationally trivial — just a threshold. The downside is the 'dying ReLU' problem: if a neuron's input is always negative, its gradient is always 0 and it can never recover. Variants like Leaky ReLU and ELU address this.
Danger Vanishing Gradient Problem
\[\boldsymbol{\delta}^{(l)} \propto \prod_{k=l}^{L-1}\sigma'(\mathbf{z}^{(k)})\]
With sigmoid: attenuation factor per layer = \((1/4)^k\)
Layers
Attenuation
5
\(9.8 \times 10^{-4}\)
10
\(9.5 \times 10^{-7}\)
20
\(9.1 \times 10^{-13}\)
After 10 layers, gradients shrink by ~1 million times . Early layers learn 1 million times slower.
Chapter 17
Full Notes →
The vanishing gradient problem is the fundamental challenge for deep networks with sigmoid/tanh activations. The error signal delta is proportional to a product of activation derivatives. Since sigmoid's maximum derivative is 1/4, each layer attenuates the gradient by at most 4x. After 10 layers, gradients are a million times smaller than at the output. Early layers essentially stop learning. This is why deep networks with sigmoid couldn't be trained until ReLU and other innovations arrived.
Solutions Fixing the Vanishing Gradient
ReLU activation: gradient = 1 for positive inputs (no saturation)Residual connections: \(\mathbf{a}^{(l)} = \sigma(\mathbf{z}^{(l)}) + \mathbf{a}^{(l-1)}\) — skip connections allow gradient to flow directlyBatch normalization: normalize pre-activations in each mini-batch, keeping signals in the non-saturating regimeCareful initialization:
Xavier: \(W \sim \mathcal{N}(0, \;2/(n_{\text{in}}+n_{\text{out}}))\)
He: \(W \sim \mathcal{N}(0,\; 2/n_{\text{in}})\)
Chapter 17
Full Notes →
Four key solutions to the vanishing gradient problem. (1) ReLU has gradient 1 for positive inputs — no saturation. (2) Residual connections (ResNets) add a skip connection that lets gradients bypass layers entirely. (3) Batch normalization keeps pre-activations in the non-saturating range by normalizing to zero mean and unit variance. (4) Xavier/He initialization sets initial weight variance to preserve signal magnitude across layers.
Comparison Activation Functions
Activation
Range
Max \(\sigma'\)
Saturates?
Era
Step
\(\{0,1\}\)
0
—
1943
Sigmoid
\((0,1)\)
\(1/4\)
Yes
1986
Tanh
\((-1,1)\)
1
Yes
~1990s
ReLU
\([0,\infty)\)
1
No
2011
GELU
\(\approx[-0.17,\infty)\)
\(\approx 1.08\)
No
2016+
Each generation solved a limitation: step → differentiable → zero-centered → non-saturating → smooth.
Chapter 17
Full Notes →
The evolution of activation functions mirrors the history of neural networks. The step function (McCulloch-Pitts) is not differentiable, so it can't be used with gradient descent. Sigmoid made backprop possible but saturates. Tanh is zero-centered but still saturates. ReLU solved saturation and enabled deep learning. GELU (used in transformers) is smooth and slightly non-monotonic, matching the needs of attention-based architectures.
Practice Gradient Checking
Numerical gradient (central differences):
\[\hat{g}_i = \frac{\mathcal{L}(\theta_i+\varepsilon) - \mathcal{L}(\theta_i-\varepsilon)}{2\varepsilon}\]
Relative error:
\[\text{rel\_error} = \frac{\|g_{\text{analytical}} - g_{\text{numerical}}\|}{\|g_{\text{analytical}}\| + \|g_{\text{numerical}}\| + \delta}\]
Pass criterion: rel_error \(< 10^{-7}\) (use \(\varepsilon \approx 10^{-7}\))
Chapter 18
Full Notes →
Gradient checking is essential for debugging backpropagation implementations. We compare the analytical gradient (from backprop) with a numerical gradient computed via central differences. The relative error should be below 10^-7 for a correct implementation. Central differences give O(epsilon^2) accuracy, much better than forward differences. Always check gradients on a small network before scaling up. Note: gradient checking is slow (O(P) forward passes) and should only be used for debugging, not training.
Demonstration XOR Network Learning
\(x_1\) \(x_2\) XOR 0 0 0 0 1 1 1 0 1 1 1 0
\(x_1\)
\(x_2\)
\(h_1\)
\(h_2\)
\(h_3\)
\(h_4\)
\(\hat{y}\)
2 → 4 → 1
Network learns hidden representations that make XOR linearly separable.
Chapter 18
Full Notes →
XOR is the canonical example that a single perceptron cannot solve but a multi-layer network can. This 2-4-1 network uses backpropagation to learn weights that make XOR linearly separable in the hidden layer's representation space. The hidden layer creates a nonlinear feature mapping where the transformed points become linearly separable, and the output layer draws the separating hyperplane.
Theorem Universal Approximation
Cybenko (1989), Hornik (1991): Let \(\sigma\) be non-constant, bounded, continuous. For any continuous \(f: [0,1]^n \to \mathbb{R}\) and any \(\varepsilon > 0\), there exist \(N\), weights, and biases such that:
\[\left|f(\mathbf{x}) - \sum_{i=1}^{N} v_i\,\sigma(\mathbf{w}_i^\top\mathbf{x}+b_i)\right| < \varepsilon\]
One hidden layer suffices — given enough neurons
Chapter 19
Full Notes →
The Universal Approximation Theorem is one of the most important theoretical results in neural network theory. It states that a single hidden layer with enough neurons can approximate any continuous function to arbitrary accuracy. This was independently proved by Cybenko (1989) for sigmoids and generalized by Hornik (1991). It justified the representational power of neural networks, but says nothing about how to find the right weights.
Intuition Bump Construction
1. Steep sigmoid ≈ step
small c
large c
2. Two steps → bump
bump
+step
-step
3. Sum of bumps ≈ f(x)
target f(x)
scaled bumps
Chapter 19
Full Notes →
This is the constructive intuition behind the Universal Approximation Theorem. Step 1: With a large weight c, a sigmoid neuron approximates a step function. Step 2: Subtracting two shifted step functions creates a bump — a localized activation. Step 3: By summing many scaled bumps at different positions, we can approximate any continuous function to arbitrary accuracy. Each bump uses 2 hidden neurons, so N bumps need 2N neurons. This proves existence but doesn't tell us how to find the weights.
Warning UAT: Guarantees vs Limitations
What UAT guarantees
One hidden layer can represent any continuous function
Works for any non-polynomial activation (ReLU, sigmoid, tanh)
What UAT does NOT tell us
Required width \(N\) (can be exponentially large)
Whether gradient descent can find the solution
Generalization to unseen data
Depth efficiency
UAT is an existence theorem , not a construction algorithm .
Chapter 19
Full Notes →
It's important to understand what the UAT does and does not say. It guarantees that a sufficiently wide single-hidden-layer network CAN represent any continuous function. But it says nothing about how many neurons are needed (could be exponential), whether GD will find the solution, or whether the result will generalize. It's a pure existence theorem — very reassuring about the representational power of neural networks, but not a practical guide to architecture design.
Theorem Depth Separation
Telgarsky (2016): Functions computable by depth \(O(k)\) networks with polynomial width may require exponential width \(2^{\Omega(k)}\) with constant depth.
Deep network with \(L\) layers and \(n\) neurons per layer: \(O(n^L)\) linear regions
Single hidden layer with \(n\) neurons: only \(O(n)\) linear regions
Depth is exponentially more efficient than width
This is why we use deep networks — depth buys representational power exponentially cheaper than width.
Chapter 19
Full Notes →
Telgarsky's depth separation theorem provides the theoretical justification for deep learning. While the UAT says one layer is enough, depth separation shows that deeper networks are exponentially more efficient. A deep ReLU network with L layers can represent functions with O(n^L) linear regions, while a shallow network needs O(n) neurons — exponentially more parameters for the same expressiveness. This is the mathematical reason why 'deep' learning works better than 'wide' learning.
Part V Recap
ERM
Learning = optimization (minimizing empirical loss over training data)
Backprop
Chain rule computes exact gradients efficiently (one backward pass)
Vanishing
Sigmoid causes exponential signal decay — early layers stop learning
ReLU
Non-saturating activation solves gradient flow (gradient = 1 for positive inputs)
UAT
One hidden layer suffices for representation — but depth is exponentially more efficient
Chapters 15–19
Full Notes →
Part V covered the complete story of training neural networks. ERM frames learning as optimization. Backpropagation makes gradient computation efficient. The vanishing gradient problem limited early deep networks. ReLU and other innovations solved it. The Universal Approximation Theorem justifies the representational power of neural networks, while depth separation explains why deeper is better.
Preview Part VI: Synthesis
Chapter 20: The complete historical arc from McCulloch-Pitts to backpropagation
Three recurring themes: Representation , Learning , Universality
Lessons from AI Winters — why progress stalled and what reignited it
From logic gates (1943) to universal function approximation (1989) — the classical foundations that made modern deep learning possible.
Preview
Full Notes →
Part VI synthesizes the entire course. We trace the arc from McCulloch and Pitts' binary neurons through the perceptron, the XOR crisis, learning rules, and backpropagation. Three themes recur: representation (what can networks compute?), learning (how do they improve?), and universality (what are their limits?). We also draw lessons from the AI Winters — periods where overpromising led to funding cuts and stalled progress.
End of Part V
From Gradient to
Next: Part VI — Synthesis (Chapter 20)
End of Part V. We've covered gradient descent, the backpropagation algorithm, activation functions, practical considerations, and the Universal Approximation Theorem. Next, Part VI brings together the entire classical foundations story.