WordGPT
โ† Math: from algebra to machine learning

The math of machine learning ยท Part 3 of 5

Backpropagation

How neural networks learn: the chain rule applied layer by layer, gradients of the loss, and gradient descent updates.

A neural network is a stack of simple functions: multiply by weights, add a bias, apply an activation. Training means finding weights that make the network's predictions match known answers. The one tool that makes this possible is backpropagation: an efficient way to compute how the loss changes when you nudge any single weight, using the chain rule from calculus.

Without backpropagation you would have to re-measure the effect of every weight separately, which is hopeless for millions of weights. With it, one pass through the network forward (to predict) and one pass backward (to propagate error signals) gives you every gradient at once. This lesson builds that idea from a single neuron up to a small two-layer network.

The setup: loss, weights, and what we want

Suppose the network takes an input and produces a prediction . We measure how wrong it is with a loss function, for example the squared error

where is the true answer. Training asks: in which direction should we change each weight to make smaller? The answer is the derivative , the gradient. If it is positive, decreasing lowers the loss; if negative, increase . Gradient descent updates each weight by

where is the learning rate, a small step size. Everything hinges on computing those derivatives efficiently, which is exactly what backpropagation does.

Interactive curve of squared-error loss versus weight w for a sigmoid neuron, with a slider for the bias b from -3 to 3; the curve is a smooth bowl-like shape whose minimum shifts as b changes.-4-22400.20.40.60.81L(w) = ยฝ(ฯƒ(w+b) โˆ’ 1)ยฒ
The loss L = ยฝ(ฯƒ(wx+b) โˆ’ 1)ยฒ as a function of the weight w, with target y = 1 and x = 1. Drag b and watch where the minimum sits: training moves w downhill toward it, and the slope at the current w is exactly the gradient โˆ‚L/โˆ‚w.

One neuron, one chain rule

Take the simplest case: a single neuron computing where , and loss . The chain rule says the derivative through the composition is the product of the local derivatives:

Each factor is easy on its own: , , and . For the sigmoid activation, has the tidy derivative . Multiply the factors and you have the gradient for ; for the last factor is instead of .

The key mental model: the network is a pipeline, and backpropagation sends the error signal backward through the pipeline, multiplying by each stage's local slope as it goes.

Two curves: the sigmoid ฯƒ(z) rising from 0 to 1, and its derivative ฯƒโ€ฒ(z) peaking at 0.25 at z = 0. A mark at z = 2 shows ฯƒ(2) โ‰ˆ 0.881 and ฯƒโ€ฒ(2) โ‰ˆ 0.105.-6-4-22460.51ฯƒ(2) โ‰ˆ 0.881ฯƒโ€ฒ(2) โ‰ˆ 0.105ฯƒ(z)ฯƒโ€ฒ(z) = ฯƒ(z)(1โˆ’ฯƒ(z))
The sigmoid ฯƒ(z) and its derivative ฯƒโ€ฒ(z) = ฯƒ(z)(1โˆ’ฯƒ(z)). At the worked example's z = 2 the slope is only about 0.105 โ€” one of the shrinking factors in the chain-rule product.

Example 1. Gradient for a single sigmoid neuron A neuron computes with , , input , and target . Using squared loss , find .

First compute the forward pass.

Now the backward pass, factor by factor.

Multiply:

The gradient is negative, so gradient descent increases , pushing the prediction toward . That is the right direction.

Two layers: chaining gradients backward

Now stack two neurons: layer 1 computes with , and layer 2 computes with . The loss is still . The gradient with respect to the early weight must pass through both layers:

Notice the structure: define the output error signal . Then the hidden-layer error signal is โ€” the output signal multiplied by the weight it traveled through and the local slope. Gradients follow immediately: and . This reuse of 's is why backpropagation is fast: each signal is computed once and shared by all weights in that layer.

Interactive curve of the gradient of the loss with respect to the first-layer weight w1, for a two-sigmoid-layer network with target 1; a slider varies the second-layer weight w2 from -3 to 3. At w1 = w2 = 1 the gradient is about -0.014.-3-2-1123-0.15-0.1-0.050.050.10.15โˆ‚L/โˆ‚wโ‚
The gradient โˆ‚L/โˆ‚wโ‚ for the two-layer network (x = 1, target y = 1) as a function of wโ‚, with a slider for the second-layer weight wโ‚‚. At wโ‚ = 1, wโ‚‚ = 1 the gradient is about โˆ’0.0140, matching the worked example; dragging wโ‚‚ shows how the signal through layer 2 scales the early gradient.

Example 2. Backpropagation through two layers With , , , sigmoid activations in both layers, and target , compute for the loss .

Forward pass.

Output error signal.

Hidden error signal.

Gradient for .

Both gradients are negative, so both weights increase, moving up toward .

The four steps of an epoch

Putting it together, one training step (on one example, or a batch) has four stages:

  1. Forward pass. Compute 's and activations layer by layer, storing each layer's output.
  2. Loss. Compare the prediction with the target.
  3. Backward pass. Start with at the output layer, then propagate it backward: each layer's is the next layer's multiplied by the connecting weights and the local activation derivative.
  4. Update. Every weight takes a step , where is the error signal arriving at that weight's input side and is the activation feeding it.

Two practical notes. First, the stored forward activations are needed in the backward pass โ€” that is why training uses more memory than prediction. Second, the derivative of the activation matters a lot: sigmoid's slope is at most , so signals shrink at every layer (the vanishing gradient problem), which is one reason modern networks use activations like ReLU, whose slope is for positive inputs.

Loss curve ยฝ(ฯƒ(w) โˆ’ 1)ยฒ versus weight w with dots marking gradient-descent iterates from w = 1 with learning rate 1; the dots step down the curve toward the minimum, with loss falling from about 0.053.0.511.522.5300.050.10.15w = 1L(w) = ยฝ(ฯƒ(w) โˆ’ 1)ยฒ
Gradient descent on the single neuron's loss with ฮท = 1, starting from w = 1 (the worked example's setup). Each dot is one update w โ† w โˆ’ ฮทยทโˆ‚L/โˆ‚w: the loss shrinks every step as the weight climbs toward the minimum near w โ‰ˆ 1.9.

Example 3. One full update step Using the two-layer network from the previous example with , , target , and learning rate , perform one gradient descent update of both weights.

From the previous example: and .

Update each weight.

Check by recomputing the prediction with the new weights.

The prediction moved from toward the target , so the step helped. Loss before: ; after: โ€” smaller, as expected.

Why the chain rule makes it cheap

A naive way to get for each weight would be finite differences: nudge by a tiny , re-run the network, and compute . That costs a full forward pass per weight. With weights, one gradient estimate costs forward passes.

Backpropagation gets all gradients for the price of about one forward pass plus one backward pass, because the error signals are shared: every weight in a layer reuses the same incoming . This is an application of reverse-mode automatic differentiation, and it is the reason training deep networks with millions of parameters is computationally feasible at all. The math never changes from what we did above โ€” only the number of chained factors grows.

Common mistakes

  • Forgetting the activation derivative when propagating the error signal, e.g. writing instead of .
  • Using the loss derivative alone as the output-layer error signal instead of multiplying by , i.e. .
  • Confusing the gradient factor for a weight (, the input feeding it) with the factor for a bias ().
  • Updating weights in the wrong direction, , which climbs the loss instead of descending it.
  • Thinking backpropagation is a different algorithm from gradient descent; backpropagation only computes the gradients, and gradient descent still performs the update.

Practice

  1. A single neuron computes , where is the sigmoid function, with weight , input , and target . Since and , compute the forward-pass value .
  2. A single neuron computes , where is the sigmoid function (so is the sigmoid output, not the linear value ), with weight , input , and target , using the loss . Since , the gradient is . Compute .
  3. A single sigmoid neuron computes with input and target , currently (so ), using the loss . The gradient is . After one gradient descent step with learning rate , what is the new value of ? Give an exact expression written in terms of (for example using exp), not a decimal.
  4. A two-layer network with sigmoid activations computes and , with , , , target , and loss . The gradient with respect to the second-layer weight is . Compute exactly, as an expression written in terms of (for example using exp), not a decimal.
  5. For the same two-layer sigmoid network (, , with , , , target , loss ), the gradient with respect to the first-layer weight is . Compute exactly, as an expression written in terms of (for example using exp), not a decimal.
Answers
  1. , since and .
  2. .
  3. The gradient is , so .
  4. With and , .
  5. .

Backpropagation tunes the weights of a fixed network architecture; in the next lesson on principal component analysis you will see how to choose the shape of the data representation itself.

Part of the free WordGPT Learn section. WordGPT is an AI writing workspace for documents, resumes, websites and study.

    Backpropagation โ€” Math: from algebra to machine learning | WordGPT