The math of machine learning ยท Part 3 of 5
Backpropagation
How neural networks learn: the chain rule applied layer by layer, gradients of the loss, and gradient descent updates.
The math of machine learning10 min read
A neural network is a stack of simple functions: multiply by weights, add a bias, apply an activation. Training means finding weights that make the network's predictions match known answers. The one tool that makes this possible is backpropagation: an efficient way to compute how the loss changes when you nudge any single weight, using the chain rule from calculus.
Without backpropagation you would have to re-measure the effect of every weight separately, which is hopeless for millions of weights. With it, one pass through the network forward (to predict) and one pass backward (to propagate error signals) gives you every gradient at once. This lesson builds that idea from a single neuron up to a small two-layer network.
The setup: loss, weights, and what we want
Suppose the network takes an input and produces a prediction . We measure how wrong it is with a loss function, for example the squared error
where is the true answer. Training asks: in which direction should we change each weight to make smaller? The answer is the derivative , the gradient. If it is positive, decreasing lowers the loss; if negative, increase . Gradient descent updates each weight by
where is the learning rate, a small step size. Everything hinges on computing those derivatives efficiently, which is exactly what backpropagation does.
One neuron, one chain rule
Take the simplest case: a single neuron computing where , and loss . The chain rule says the derivative through the composition is the product of the local derivatives:
Each factor is easy on its own: , , and . For the sigmoid activation, has the tidy derivative . Multiply the factors and you have the gradient for ; for the last factor is instead of .
The key mental model: the network is a pipeline, and backpropagation sends the error signal backward through the pipeline, multiplying by each stage's local slope as it goes.
Example 1. Gradient for a single sigmoid neuron A neuron computes with , , input , and target . Using squared loss , find .
First compute the forward pass.
Now the backward pass, factor by factor.
Multiply:
The gradient is negative, so gradient descent increases , pushing the prediction toward . That is the right direction.
Two layers: chaining gradients backward
Now stack two neurons: layer 1 computes with , and layer 2 computes with . The loss is still . The gradient with respect to the early weight must pass through both layers:
Notice the structure: define the output error signal . Then the hidden-layer error signal is โ the output signal multiplied by the weight it traveled through and the local slope. Gradients follow immediately: and . This reuse of 's is why backpropagation is fast: each signal is computed once and shared by all weights in that layer.
Example 2. Backpropagation through two layers With , , , sigmoid activations in both layers, and target , compute for the loss .
Forward pass.
Output error signal.
Hidden error signal.
Gradient for .
Both gradients are negative, so both weights increase, moving up toward .
The four steps of an epoch
Putting it together, one training step (on one example, or a batch) has four stages:
- Forward pass. Compute 's and activations layer by layer, storing each layer's output.
- Loss. Compare the prediction with the target.
- Backward pass. Start with at the output layer, then propagate it backward: each layer's is the next layer's multiplied by the connecting weights and the local activation derivative.
- Update. Every weight takes a step , where is the error signal arriving at that weight's input side and is the activation feeding it.
Two practical notes. First, the stored forward activations are needed in the backward pass โ that is why training uses more memory than prediction. Second, the derivative of the activation matters a lot: sigmoid's slope is at most , so signals shrink at every layer (the vanishing gradient problem), which is one reason modern networks use activations like ReLU, whose slope is for positive inputs.
Example 3. One full update step Using the two-layer network from the previous example with , , target , and learning rate , perform one gradient descent update of both weights.
From the previous example: and .
Update each weight.
Check by recomputing the prediction with the new weights.
The prediction moved from toward the target , so the step helped. Loss before: ; after: โ smaller, as expected.
Why the chain rule makes it cheap
A naive way to get for each weight would be finite differences: nudge by a tiny , re-run the network, and compute . That costs a full forward pass per weight. With weights, one gradient estimate costs forward passes.
Backpropagation gets all gradients for the price of about one forward pass plus one backward pass, because the error signals are shared: every weight in a layer reuses the same incoming . This is an application of reverse-mode automatic differentiation, and it is the reason training deep networks with millions of parameters is computationally feasible at all. The math never changes from what we did above โ only the number of chained factors grows.
Common mistakes
- Forgetting the activation derivative when propagating the error signal, e.g. writing instead of .
- Using the loss derivative alone as the output-layer error signal instead of multiplying by , i.e. .
- Confusing the gradient factor for a weight (, the input feeding it) with the factor for a bias ().
- Updating weights in the wrong direction, , which climbs the loss instead of descending it.
- Thinking backpropagation is a different algorithm from gradient descent; backpropagation only computes the gradients, and gradient descent still performs the update.
Practice
- A single neuron computes , where is the sigmoid function, with weight , input , and target . Since and , compute the forward-pass value .
- A single neuron computes , where is the sigmoid function (so is the sigmoid output, not the linear value ), with weight , input , and target , using the loss . Since , the gradient is . Compute .
- A single sigmoid neuron computes with input and target , currently (so ), using the loss . The gradient is . After one gradient descent step with learning rate , what is the new value of ? Give an exact expression written in terms of (for example using exp), not a decimal.
- A two-layer network with sigmoid activations computes and , with , , , target , and loss . The gradient with respect to the second-layer weight is . Compute exactly, as an expression written in terms of (for example using exp), not a decimal.
- For the same two-layer sigmoid network (, , with , , , target , loss ), the gradient with respect to the first-layer weight is . Compute exactly, as an expression written in terms of (for example using exp), not a decimal.
Answers
- , since and .
- .
- The gradient is , so .
- With and , .
- .
Backpropagation tunes the weights of a fixed network architecture; in the next lesson on principal component analysis you will see how to choose the shape of the data representation itself.