Lesson 22 of 48
The chain rule
The one rule every neural network is built out of.
Composition is how you build a deep function; the chain rule is how you differentiate one. Backpropagation is not an algorithm so much as this rule applied bookkeepingly at scale.
Do this
Differentiate a three-deep composition by hand, then verify numerically at a specific point — they must agree to at least four decimal places.
The question that unlocks the next lesson
For f(g(x)), the derivative is…
- Af′(g(x))
- Bf′(g(x)) · g′(x)
- Cf′(x) · g′(x)
- Dg′(f(x)) · f′(x)