Skip to main content

Ctrl+K

Classical Foundations of Artificial Neural Networks

Classical Foundations of Artificial Neural Networks

Part I: Origins

Chapter 1: The McCulloch-Pitts Neuron (1943)
Chapter 2: Building Logic from Neurons
Chapter 3: Historical Context and Timeline

Part II: The Perceptron

Chapter 4: Rosenblatt’s Perceptron
Chapter 5: The Perceptron Learning Algorithm
Chapter 6: The Perceptron Convergence Theorem
Chapter 7: What Perceptrons Can Compute

Part III: Limitations and Breakthroughs

Chapter 8: The XOR Problem
Chapter 9: Minsky and Papert’s Analysis
Chapter 10: Linear Separability — A Deep Dive
Chapter 11: Breaking Through — Multi-Layer Networks

Part IV: Learning Rules

Chapter 12: Hebbian Learning Theory
Chapter 13: Oja’s Rule and Principal Component Analysis
Chapter 14: The Zoo of Learning Rules

Part V: Backpropagation

Chapter 15: Gradient Descent Foundations
Chapter 16: Backpropagation – The Complete Derivation
Chapter 17: Activation Functions and the Vanishing Gradient
Chapter 18: Implementing Backpropagation from Scratch
Chapter 19: The Universal Approximation Theorem

Part VI: Synthesis

Chapter 20: From McCulloch-Pitts to Backpropagation – The Complete Arc
Appendix: Complete Bibliography and Reading Guide

Part VII: Convolutional Neural Networks

Chapter 21: From Pixels to Features
Chapter 22: The Convolution Operation
Chapter 23: Building a CNN from Scratch
Chapter 24: Training a CNN with Backpropagation
Chapter 25: CNN Experiments and Analysis

Part VIII: Modern Optimization

Chapter 26: Cross-Entropy, KL Divergence, and Maximum Likelihood
Chapter 27: Momentum, RMSProp, and Adam
Chapter 28: Autograd – Automatic Differentiation

Part IX: Introduction to PyTorch

Chapter 29: From Micrograd to PyTorch – Tensors, Autograd, and nn.Module
Chapter 30: DataLoaders, Training Loops, and Evaluation
Chapter 31: PyTorch CNN on MNIST

Part X: Recurrent Neural Networks & LSTM

Chapter 32: Sequences, Time, and Memory
Chapter 33: Backpropagation Through Time
Chapter 34: LSTM — The Gating Revolution
Chapter 35: Character-Level Language Modeling
Chapter 36: Sequence-to-Sequence and Beyond

Part XI: Attention & Transformers

Chapter 37: Attention — Looking Back
Chapter 38: Attention Variants
Chapter 39: Self-Attention
Chapter 40: Attention Is All You Need

Self-Assessment

Test: Review and Self-Assessment

Reference

PyTorch Cheat Sheet

Interactive Papers

Interactive Papers
Overview & Notations
Three Separation Lemmas
The Universal Approximation Theorem
Extensions, Connections & Challenges

Resources

Lecture Slides
Errata

Search

Ctrl+K

By Bartosz Naskręcki

© Copyright 2026 Bartosz Naskręcki (co-generated with Claude).