Part V

# Backpropagation

### The Algorithm That Changed Everything

Chapters 15–19

Part V covers the mathematical engine behind modern neural networks. We start with gradient descent as an optimization framework, derive the backpropagation equations from the chain rule, analyze activation functions and the vanishing gradient problem, discuss practical considerations, and conclude with the Universal Approximation Theorem.

## Definition Empirical Risk Minimization

ERM objective: find parameters that minimize average loss over training data: \[\min_{\mathbf{w}} \; \mathcal{L}(\mathbf{w}) = \frac{1}{N} \sum_{i=1}^{N} L\bigl(f(\mathbf{x}^{(i)};\mathbf{w}),\, \mathbf{y}^{(i)}\bigr)\]

- \(\mathbf{w}\): learnable parameters (weights + biases)
- \(L\): per-sample loss (measures prediction error)
- Learning = finding \(\mathbf{w}^*\) that minimizes average loss

Chapter 15 Full Notes →
The ERM framework is the bridge between machine learning and optimization. Instead of hand-crafting rules, we define a loss function and let an algorithm find the best parameters. The loss L measures how far the prediction f(x; w) is from the true label y. We average over all N training samples to get the empirical risk. This reformulates LEARNING as OPTIMIZATION.

## Algorithm Gradient Descent

\[\mathbf{w}_{t+1} = \mathbf{w}_t - \eta \,\nabla\mathcal{L}(\mathbf{w}_t)\]

- Gradient \(\nabla\mathcal{L}\) points in direction of steepest ascent
- Negative gradient \(-\nabla\mathcal{L}\) gives steepest descent direction

Chapter 15 Full Notes →
This is the core update rule of gradient descent. At each step t, we compute the gradient of the loss at the current parameters, multiply by the learning rate eta, and subtract from the current parameters. The gradient points uphill, so the negative gradient points downhill — toward lower loss. The learning rate eta controls step size.

## Geometry Gradient & Contour Lines

Chapter 15 Full Notes →
This contour plot shows a 2D loss surface. Each ellipse represents a set of parameter values with the same loss. The gradient at any point is perpendicular to the contour line and points toward higher loss (red arrow). The negative gradient points toward lower loss (green arrow). The GD path zigzags because the gradient direction changes at each step. Note how the path oscillates across the narrow valley — this motivates momentum-based methods.

## Hyperparameter Learning Rate Effects

Chapter 15 Full Notes →
The learning rate eta is the most critical hyperparameter. Too small: convergence is painfully slow, many steps barely move toward the minimum. Just right: smooth convergence in a reasonable number of steps. Too large: the algorithm overshoots the minimum, oscillates wildly, and can diverge to infinity. Finding the right learning rate is often the first tuning step in practice.

## Landscape Convex vs Non-Convex

Convex

- Unique global minimum
- GD converges for proper \(\eta\)

Non-Convex

- Multiple local minima
- Saddle points
- Neural networks are non-convex

Chapter 15 Full Notes →
Convex loss surfaces have a single global minimum — GD is guaranteed to find it with a proper learning rate. Non-convex surfaces have multiple local minima and saddle points. Neural network loss landscapes are highly non-convex, so GD may converge to local minima or get stuck at saddle points. In practice, SGD with momentum often works well despite non-convexity.

## Definition Loss Functions

Mean Squared Error
\[L_{\text{MSE}} = \frac{1}{2}\|\hat{\mathbf{y}} - \mathbf{y}\|^2\] Gradient: \(\hat{\mathbf{y}} - \mathbf{y}\)
Cross-Entropy
\[L_{\text{CE}} = -\sum_j y_j \log \hat{y}_j\] Gradient with softmax: \(\hat{y}_j - y_j\)
Both gradients have the elegant form: prediction − target. Error drives learning.
Chapter 15 Full Notes →
The two most common loss functions. MSE is used for regression; the 1/2 factor simplifies the derivative. Cross-entropy is used for classification, especially with softmax output. Both produce gradients that are just prediction minus target — a beautifully simple and interpretable error signal that drives learning.

## Comparison Batch vs SGD vs Mini-batch

Method | Cost/iteration | Convergence | Notes
--- | --- | --- | ---
Batch GD | \(O(N)\) | \(O(1/t)\) | Smooth, exact gradient
SGD | \(O(1)\) | \(O(1/\sqrt{t})\) | Fast updates, noisy
Mini-batch | \(O(B)\) | Between | Practical standard

Mini-batch (B = 32–256) balances computational efficiency and gradient quality.
Chapter 15 Full Notes →
Batch GD uses all N samples per update — exact gradient but expensive. SGD uses a single sample — very fast but noisy. Mini-batch SGD is the practical standard, using B samples per update (typically 32-256). The noise in SGD/mini-batch can actually help escape local minima. Modern training uses mini-batch with adaptive learning rates (Adam).

## From Optimization to Networks

Now we have gradient descent — but how do we compute gradients through a multi-layer network?

We've established the optimization framework: minimize a loss function using gradient descent. But for neural networks with multiple layers, computing the gradient is not trivial. We need the chain rule applied systematically — this is backpropagation.

## Notation Network Architecture

Chapter 16 Full Notes →
The standard notation for a multi-layer network. Superscript (l) denotes the layer. a^(0) is the input, z^(l) is the pre-activation (linear combination), a^(l) is the post-activation. Weight matrix W^(l) connects layer l-1 to layer l. This notation makes the backpropagation equations clean and general.

## Algorithm Forward Pass

\[\mathbf{z}^{(l)} = \mathbf{W}^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}\]
\[\mathbf{a}^{(l)} = \sigma^{(l)}\bigl(\mathbf{z}^{(l)}\bigr)\]

- Pre-activation \(\mathbf{z}\): linear transformation (matrix multiply + bias)
- Post-activation \(\mathbf{a}\): nonlinear squashing (sigmoid, ReLU, etc.)

Chapter 16 Full Notes →
The forward pass computes the output layer by layer, from input to output. At each layer l, we first compute the pre-activation z as a linear combination of the previous layer's activations, then apply the nonlinear activation function sigma. This separation into linear and nonlinear steps is key to the clean derivation of backpropagation. All intermediate values z and a are cached for the backward pass.

## Theorem Chain Rule

The multivariate chain rule for composed functions: \[\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \frac{\partial \mathcal{L}}{\partial z_i^{(l)}} \cdot \frac{\partial z_i^{(l)}}{\partial w_{ij}^{(l)}} = \delta_i^{(l)} \cdot a_j^{(l-1)}\]
Key insight: define the error signal \(\delta_i^{(l)} = \partial\mathcal{L}/\partial z_i^{(l)}\) — the sensitivity of the loss to each pre-activation.

- Once we know all \(\delta\) values, weight gradients follow immediately
- The challenge: computing \(\delta\) efficiently layer by layer

Chapter 16 Full Notes →
The chain rule is the mathematical foundation of backpropagation. We decompose the partial derivative of the loss with respect to each weight into two factors: the error signal delta (how sensitive the loss is to the pre-activation) and the input activation (from the previous layer). The delta values are the key quantities — once we have them, all gradients follow by simple multiplication.

## BP1 Output Layer Error

\[\boxed{\boldsymbol{\delta}^{(L)} = \nabla_{\mathbf{a}^{(L)}}\mathcal{L} \odot \sigma'^{(L)}\bigl(\mathbf{z}^{(L)}\bigr)}\]

- First factor: how loss changes with output activation
- Second factor: how activation changes with pre-activation (\(\odot\) = element-wise product)

This is the starting point of the backward pass — error begins at the output.
Chapter 16 Full Notes →
BP1 is the first backpropagation equation. It computes the error signal at the output layer L. The first factor is the derivative of the loss with respect to the output activations — for MSE this is (y_hat - y). The second factor is the derivative of the activation function evaluated at the pre-activation z. The Hadamard product combines them element-wise. This is where error information originates.

## BP2 Error Backpropagation

\[\boxed{\boldsymbol{\delta}^{(l)} = \bigl((\mathbf{W}^{(l+1)})^\top \boldsymbol{\delta}^{(l+1)}\bigr) \odot \sigma'^{(l)}\bigl(\mathbf{z}^{(l)}\bigr)}\]

- Error propagates backward through transpose weights
- Each layer's error depends on the next layer's error (recursive)

The transpose \((\mathbf{W}^{(l+1)})^\top\) distributes error back to contributing neurons.
Chapter 16 Full Notes →
BP2 is the recursive step — the heart of backpropagation. It expresses the error at layer l in terms of the error at layer l+1. The transpose weight matrix distributes error from the next layer back to the current layer's neurons, proportional to how much each neuron contributed. The sigma' factor accounts for how much the activation function attenuated the signal. This recursion runs backward from the output to the first hidden layer.

## BP3 + BP4 Weight & Bias Gradients

\[\boxed{\frac{\partial\mathcal{L}}{\partial\mathbf{W}^{(l)}} = \boldsymbol{\delta}^{(l)}\bigl(\mathbf{a}^{(l-1)}\bigr)^\top}\]
\[\boxed{\frac{\partial\mathcal{L}}{\partial\mathbf{b}^{(l)}} = \boldsymbol{\delta}^{(l)}}\]

- BP3: weight gradient = error \(\times\) input (outer product)
- BP4: bias gradient = error signal directly

Chapter 16 Full Notes →
BP3 and BP4 convert error signals into parameter gradients. BP3 says the weight gradient is the outer product of the error signal delta and the input activations from the previous layer. This is a Hebbian-like rule: weights change proportionally to how much the input contributed and how wrong the output was. BP4 is even simpler: the bias gradient equals the error signal directly, since the bias always has input 1.

## Reference The Four BP Equations

# | Name | Equation | Purpose
--- | --- | --- | ---
BP1 | Output Error | \(\boldsymbol{\delta}^{(L)} = \nabla_{\mathbf{a}}\mathcal{L} \odot \sigma'(\mathbf{z}^{(L)})\) | Start the backward pass
BP2 | Error Backprop | \(\boldsymbol{\delta}^{(l)} = (W^{(l+1)})^\top\boldsymbol{\delta}^{(l+1)} \odot \sigma'(\mathbf{z}^{(l)})\) | Propagate to earlier layers
BP3 | Weight Gradient | \(\partial\mathcal{L}/\partial W^{(l)} = \boldsymbol{\delta}^{(l)}(\mathbf{a}^{(l-1)})^\top\) | How to update weights
BP4 | Bias Gradient | \(\partial\mathcal{L}/\partial \mathbf{b}^{(l)} = \boldsymbol{\delta}^{(l)}\) | How to update biases

These four equations are all you need. Everything else is bookkeeping.
Chapter 16 Full Notes →
This is the complete reference for the four backpropagation equations. BP1 initializes the error at the output. BP2 propagates it backward layer by layer. BP3 and BP4 convert errors into parameter gradients. Together, they allow us to compute the gradient of the loss with respect to every parameter in the network in a single backward pass. This is the algorithm that made deep learning practical.

## Algorithm Backpropagation Pseudocode

BACKPROPAGATION(X, Y, network): // Forward pass for l = 1 to L: z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾ a⁽ˡ⁾ = σ⁽ˡ⁾(z⁽ˡ⁾) // Backward pass δ⁽ᴸ⁾ = ∇ₐL ⊙ σ'(z⁽ᴸ⁾) // BP1 for l = L-1 down to 1: δ⁽ˡ⁾ = (W⁽ˡ⁺¹⁾)ᵀδ⁽ˡ⁺¹⁾ ⊙ σ'(z⁽ˡ⁾) // BP2 // Update for l = 1 to L: W⁽ˡ⁾ -= η · δ⁽ˡ⁾(a⁽ˡ⁻¹⁾)ᵀ // BP3 b⁽ˡ⁾ -= η · δ⁽ˡ⁾ // BP4
Chapter 16 Full Notes →
The complete backpropagation algorithm in pseudocode. Three phases: (1) Forward pass — compute all z and a values layer by layer. (2) Backward pass — compute error signals delta from output to input using BP1 and BP2. (3) Update — use BP3 and BP4 to adjust weights and biases. This entire procedure is repeated for each mini-batch until convergence.

## Insight Computational Complexity

Backpropagation cost:

- Forward pass: \(O(L \cdot n^2)\) for \(L\) layers of width \(n\)
- Backward pass: same complexity (~2× forward cost)

Alternative: finite differences would need \(O(P)\) forward passes (one per parameter).
For 1M parameters: backprop is ~500,000× faster.
Backprop trades memory (storing all activations) for speed (one backward pass).
Chapter 16 Full Notes →
The efficiency of backpropagation is what makes neural networks trainable. Computing gradients via finite differences would require 2P forward passes (two per parameter), which is prohibitively expensive for networks with millions of parameters. Backpropagation computes ALL gradients in a single backward pass, at roughly twice the cost of a single forward pass. The tradeoff is memory: we must store all intermediate activations for the backward pass.

## History Backpropagation Timeline

Year | Contributors | Contribution
--- | --- | ---
1970 | Linnainmaa | Reverse-mode automatic differentiation
1974 | Werbos | Applied to neural networks (PhD thesis)
1982 | Hopfield | Revived neural networks via physics
1986 | Rumelhart, Hinton & Williams | Nature paper popularizing backprop

The algorithm was discovered multiple times independently — but the 1986 paper gave the community a practical tool and clear exposition.
Chapter 16 Full Notes →
Backpropagation has a rich history of independent rediscovery. Linnainmaa described the mathematical core (reverse-mode AD) in 1970. Werbos applied it to neural networks in 1974, but his work wasn't widely known. After Hopfield revived interest in neural networks, Rumelhart, Hinton, and Williams published their seminal 1986 Nature paper that made backpropagation the standard training algorithm.

## Definition Sigmoid Activation

\[\sigma(x) = \frac{1}{1+e^{-x}}\] \[\sigma'(x) = \sigma(x)(1-\sigma(x))\]

- Range: \((0,1)\)
- Max derivative = \(1/4\) at \(x=0\)
- Saturates at extremes

Chapter 17 Full Notes →
The sigmoid function was the standard activation in early neural networks. It maps any real number to (0,1), making it interpretable as a probability. Its derivative has the elegant form sigma(x)(1-sigma(x)), which is maximal at x=0 (value 1/4) and approaches 0 at the extremes. This saturation is a key weakness: for large |x|, the gradient nearly vanishes.

## Definition Tanh Activation

\[\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}\] \[\tanh'(x) = 1 - \tanh^2(x)\]

- Range: \((-1,1)\)
- Zero-centered (unlike sigmoid)
- Max derivative = 1 at \(x=0\)

Chapter 17 Full Notes →
Tanh is a rescaled sigmoid: tanh(x) = 2*sigma(2x) - 1. Its key advantage is being zero-centered, which helps gradient descent converge faster because the outputs are balanced around 0. The maximum derivative is 1 (vs 1/4 for sigmoid), which reduces gradient attenuation. However, tanh still saturates at extremes, so it still suffers from the vanishing gradient problem for deep networks.

## Definition ReLU Activation

\[\text{ReLU}(x) = \max(0,x)\] \[\text{ReLU}'(x) = \begin{cases}1 & x > 0 \\ 0 & x \leq 0\end{cases}\]

- Non-saturating (gradient = 1 for \(x>0\))
- Computationally cheap
- Problem: dying ReLU (neurons stuck at 0)

Chapter 17 Full Notes →
ReLU (Rectified Linear Unit) was the breakthrough activation function. For positive inputs, the gradient is exactly 1, solving the vanishing gradient problem. It's also computationally trivial — just a threshold. The downside is the 'dying ReLU' problem: if a neuron's input is always negative, its gradient is always 0 and it can never recover. Variants like Leaky ReLU and ELU address this.

## Danger Vanishing Gradient Problem

\[\boldsymbol{\delta}^{(l)} \propto \prod_{k=l}^{L-1}\sigma'(\mathbf{z}^{(k)})\]

With sigmoid: attenuation factor per layer = \((1/4)^k\)

Layers | Attenuation
--- | ---
5 | \(9.8 \times 10^{-4}\)
10 | \(9.5 \times 10^{-7}\)
20 | \(9.1 \times 10^{-13}\)

After 10 layers, gradients shrink by ~1 million times. Early layers learn 1 million times slower.
Chapter 17 Full Notes →
The vanishing gradient problem is the fundamental challenge for deep networks with sigmoid/tanh activations. The error signal delta is proportional to a product of activation derivatives. Since sigmoid's maximum derivative is 1/4, each layer attenuates the gradient by at most 4x. After 10 layers, gradients are a million times smaller than at the output. Early layers essentially stop learning. This is why deep networks with sigmoid couldn't be trained until ReLU and other innovations arrived.

## Solutions Fixing the Vanishing Gradient

- ReLU activation: gradient = 1 for positive inputs (no saturation)
- Residual connections: \(\mathbf{a}^{(l)} = \sigma(\mathbf{z}^{(l)}) + \mathbf{a}^{(l-1)}\) — skip connections allow gradient to flow directly
- Batch normalization: normalize pre-activations in each mini-batch, keeping signals in the non-saturating regime
- Careful initialization:
  - Xavier: \(W \sim \mathcal{N}(0, \;2/(n_{\text{in}}+n_{\text{out}}))\)
  - He: \(W \sim \mathcal{N}(0,\; 2/n_{\text{in}})\)

Chapter 17 Full Notes →
Four key solutions to the vanishing gradient problem. (1) ReLU has gradient 1 for positive inputs — no saturation. (2) Residual connections (ResNets) add a skip connection that lets gradients bypass layers entirely. (3) Batch normalization keeps pre-activations in the non-saturating range by normalizing to zero mean and unit variance. (4) Xavier/He initialization sets initial weight variance to preserve signal magnitude across layers.

## Comparison Activation Functions

Activation | Range | Max \(\sigma'\) | Saturates? | Era
--- | --- | --- | --- | ---
Step | \(\{0,1\}\) | 0 | — | 1943
Sigmoid | \((0,1)\) | \(1/4\) | Yes | 1986
Tanh | \((-1,1)\) | 1 | Yes | ~1990s
ReLU | \([0,\infty)\) | 1 | No | 2011
GELU | \(\approx[-0.17,\infty)\) | \(\approx 1.08\) | No | 2016+

Each generation solved a limitation: step → differentiable → zero-centered → non-saturating → smooth.
Chapter 17 Full Notes →
The evolution of activation functions mirrors the history of neural networks. The step function (McCulloch-Pitts) is not differentiable, so it can't be used with gradient descent. Sigmoid made backprop possible but saturates. Tanh is zero-centered but still saturates. ReLU solved saturation and enabled deep learning. GELU (used in transformers) is smooth and slightly non-monotonic, matching the needs of attention-based architectures.

## Practice Gradient Checking

Numerical gradient (central differences): \[\hat{g}_i = \frac{\mathcal{L}(\theta_i+\varepsilon) - \mathcal{L}(\theta_i-\varepsilon)}{2\varepsilon}\]
Relative error:
\[\text{rel\_error} = \frac{\|g_{\text{analytical}} - g_{\text{numerical}}\|}{\|g_{\text{analytical}}\| + \|g_{\text{numerical}}\| + \delta}\]
Pass criterion: rel_error \(< 10^{-7}\) (use \(\varepsilon \approx 10^{-7}\))
Chapter 18 Full Notes →
Gradient checking is essential for debugging backpropagation implementations. We compare the analytical gradient (from backprop) with a numerical gradient computed via central differences. The relative error should be below 10^-7 for a correct implementation. Central differences give O(epsilon^2) accuracy, much better than forward differences. Always check gradients on a small network before scaling up. Note: gradient checking is slow (O(P) forward passes) and should only be used for debugging, not training.

## Demonstration XOR Network Learning

\(x_1\) | \(x_2\) | XOR
--- | --- | ---
0 | 0 | 0
0 | 1 | 1
1 | 0 | 1
1 | 1 | 0

Network learns hidden representations that make XOR linearly separable.
Chapter 18 Full Notes →
XOR is the canonical example that a single perceptron cannot solve but a multi-layer network can. This 2-4-1 network uses backpropagation to learn weights that make XOR linearly separable in the hidden layer's representation space. The hidden layer creates a nonlinear feature mapping where the transformed points become linearly separable, and the output layer draws the separating hyperplane.

## Theorem Universal Approximation

Cybenko (1989), Hornik (1991): Let \(\sigma\) be non-constant, bounded, continuous. For any continuous \(f: [0,1]^n \to \mathbb{R}\) and any \(\varepsilon > 0\), there exist \(N\), weights, and biases such that: \[\left|f(\mathbf{x}) - \sum_{i=1}^{N} v_i\,\sigma(\mathbf{w}_i^\top\mathbf{x}+b_i)\right| < \varepsilon\]

- One hidden layer suffices — given enough neurons

Chapter 19 Full Notes →
The Universal Approximation Theorem is one of the most important theoretical results in neural network theory. It states that a single hidden layer with enough neurons can approximate any continuous function to arbitrary accuracy. This was independently proved by Cybenko (1989) for sigmoids and generalized by Hornik (1991). It justified the representational power of neural networks, but says nothing about how to find the right weights.

## Intuition Bump Construction

Chapter 19 Full Notes →
This is the constructive intuition behind the Universal Approximation Theorem. Step 1: With a large weight c, a sigmoid neuron approximates a step function. Step 2: Subtracting two shifted step functions creates a bump — a localized activation. Step 3: By summing many scaled bumps at different positions, we can approximate any continuous function to arbitrary accuracy. Each bump uses 2 hidden neurons, so N bumps need 2N neurons. This proves existence but doesn't tell us how to find the weights.

## Warning UAT: Guarantees vs Limitations

What UAT guarantees

- One hidden layer can represent any continuous function
- Works for any non-polynomial activation (ReLU, sigmoid, tanh)

What UAT does NOT tell us

- Required width \(N\) (can be exponentially large)
- Whether gradient descent can find the solution
- Generalization to unseen data
- Depth efficiency

UAT is an existence theorem, not a construction algorithm.
Chapter 19 Full Notes →
It's important to understand what the UAT does and does not say. It guarantees that a sufficiently wide single-hidden-layer network CAN represent any continuous function. But it says nothing about how many neurons are needed (could be exponential), whether GD will find the solution, or whether the result will generalize. It's a pure existence theorem — very reassuring about the representational power of neural networks, but not a practical guide to architecture design.

## Theorem Depth Separation

Telgarsky (2016): Functions computable by depth \(O(k)\) networks with polynomial width may require exponential width \(2^{\Omega(k)}\) with constant depth.

- Deep network with \(L\) layers and \(n\) neurons per layer: \(O(n^L)\) linear regions
- Single hidden layer with \(n\) neurons: only \(O(n)\) linear regions
- Depth is exponentially more efficient than width

This is why we use deep networks — depth buys representational power exponentially cheaper than width.
Chapter 19 Full Notes →
Telgarsky's depth separation theorem provides the theoretical justification for deep learning. While the UAT says one layer is enough, depth separation shows that deeper networks are exponentially more efficient. A deep ReLU network with L layers can represent functions with O(n^L) linear regions, while a shallow network needs O(n) neurons — exponentially more parameters for the same expressiveness. This is the mathematical reason why 'deep' learning works better than 'wide' learning.

## Part V Recap

ERM Learning = optimization (minimizing empirical loss over training data)
Backprop Chain rule computes exact gradients efficiently (one backward pass)
Vanishing Sigmoid causes exponential signal decay — early layers stop learning
ReLU Non-saturating activation solves gradient flow (gradient = 1 for positive inputs)
UAT One hidden layer suffices for representation — but depth is exponentially more efficient
Chapters 15–19 Full Notes →
Part V covered the complete story of training neural networks. ERM frames learning as optimization. Backpropagation makes gradient computation efficient. The vanishing gradient problem limited early deep networks. ReLU and other innovations solved it. The Universal Approximation Theorem justifies the representational power of neural networks, while depth separation explains why deeper is better.

## Preview Part VI: Synthesis

- Chapter 20: The complete historical arc from McCulloch-Pitts to backpropagation
- Three recurring themes: Representation, Learning, Universality
- Lessons from AI Winters — why progress stalled and what reignited it

From logic gates (1943) to universal function approximation (1989) — the classical foundations that made modern deep learning possible.
Preview Full Notes →
Part VI synthesizes the entire course. We trace the arc from McCulloch and Pitts' binary neurons through the perceptron, the XOR crisis, learning rules, and backpropagation. Three themes recur: representation (what can networks compute?), learning (how do they improve?), and universality (what are their limits?). We also draw lessons from the AI Winters — periods where overpromising led to funding cuts and stalled progress.
End of Part V

# From Gradient to
Universal Approximation

### Next: Part VI — Synthesis (Chapter 20)

End of Part V. We've covered gradient descent, the backpropagation algorithm, activation functions, practical considerations, and the Universal Approximation Theorem. Next, Part VI brings together the entire classical foundations story.