WordGPT
← Math: from algebra to machine learning

The math of machine learning · Part 1 of 5

Gradient descent

How gradient descent walks downhill on a loss function, one small step at a time, to find minimum values.

Suppose you are standing on a foggy hillside and want to reach the valley bottom. You cannot see far, but you can feel the slope under your feet. A sensible strategy: check which direction points downhill, take a small step that way, repeat. Eventually the ground flattens and you stop.

Gradient descent is exactly this strategy written in mathematics, and it is the engine behind training almost every machine learning model. The hill is a loss function measuring how wrong the model is, and the valley bottom is the set of parameters that makes the model as accurate as possible. This lesson turns the hillside intuition into a precise, computable rule.

The gradient: slope as a vector

For a function of one variable, , the derivative tells you the slope at : positive means the function is increasing, negative means decreasing. For a function of several variables, , each partial derivative gives the slope in one direction. Collect them into a vector called the gradient:

The gradient points in the direction of steepest ascent — the fastest way uphill. Its negative, , points steepest downhill. That single fact is all gradient descent needs.

Example. Let . The derivative is . At the slope is , positive, so the function is climbing there — to go downhill we should move in the negative direction, toward smaller .

Parabola y = x² + 3x on x from -4 to 2, with a mark at the point (3, 18) where the slope is 9.-4-3-2-11251015x = 3, slope 9f(x) = x² + 3x
The loss f(x) = x² + 3x climbs steeply at x = 3, where the slope is 9, so a downhill step must move left toward smaller x.

Example 1. Computing a derivative for descent Find the derivative of and evaluate it at .

Differentiate term by term:

Substitute :

The slope is , so at the function rises steeply and a downhill step must decrease .

The update rule

Gradient descent repeats one simple move. If the current point is and the slope there is , step against the slope, scaled by a positive number called the learning rate:

Why subtract? If the slope is positive, the function grows to the right, so we step left; if the slope is negative, we step right. Either way, subtracting the slope moves us downhill. The learning rate controls the step size: too small and descent is painfully slow, too large and we overshoot the valley and may bounce forever or diverge.

Example. Minimize starting from with learning rate . The gradient is , so at it equals .

One more step: the slope at is , so

The iterates shrink toward , which is indeed the minimum of . Notice the steps get smaller automatically as the slope flattens near the bottom.

Parabola y = x² with marks at the first three gradient descent iterates (3, 9), (2.4, 5.76) and (1.92, 3.6864), approaching the minimum at 0.123246810x₀ = 3x₁ = 2.4x₂ = 1.92f(x) = x²
Descending f(x) = x² from x = 3 with learning rate 0.1: the iterates 3, 2.4, 1.92 land ever closer to the minimum at 0, and the steps shrink as the slope flattens.

Example 2. Two steps of gradient descent by hand Minimize by gradient descent starting at with learning rate . Compute and .

The gradient is .

At : slope .

At : slope .

Check: , so the loss really is decreasing each step.

Descent in several variables

With several parameters the rule is identical, just applied component by component: each coordinate moves against its own partial derivative. For with learning rate :

In machine learning the "coordinates" are the model's weights, often thousands or millions of them, but the update is still this same componentwise subtraction.

Rotatable 3D surface of the bowl z = x² + y² over x and y from -3 to 3, with minimum at the origin.
The loss f(x, y) = x² + y² is a bowl; gradient descent moves each coordinate against its own partial derivative, so from (1, 2) one step lands at (0.8, 1.6), lower on the bowl. Drag to view the surface from different angles.

Example 3. A step in two dimensions Let . Starting from the point with learning rate , compute one gradient descent step.

The partial derivatives are:

At the gradient is .

Update each coordinate:

Check: , so the step went downhill.

Where the walk stops

Gradient descent stops (in theory) when the gradient is zero: no slope means flat ground. Setting every partial derivative to zero and solving gives the critical points — candidates for minima, maxima, or saddle points. For a convex bowl like or a linear regression loss, the only critical point is the global minimum, so descent cannot fail to find it.

Example. Minimize by finding where the gradient vanishes.

Set it to zero:

Check by substituting back: , so the function is a bowl and is the minimum. Indeed , and nearby values like and are both larger.

In real problems the gradient is never exactly zero, so we iterate until the steps or the loss stop improving. Also, on non-convex surfaces (like deep neural networks) there can be many valleys; gradient descent finds a local minimum, not necessarily the best one — usually good enough in practice.

Parabola y = x² − 6x + 1 with its vertex marked at (3, −8), the point where the derivative 2x − 6 vanishes.246-55minimum (3, −8)f(x) = x² − 6x + 1
Setting the gradient of f(x) = x² − 6x + 1 to zero gives x = 3; the bowl shape (f'' = 2 > 0) confirms it is the minimum, with f(3) = −8.

Example 4. Finding the critical point and verifying it Find the minimum of .

Differentiate:

Set the gradient to zero:

Check by substituting back into the original function:

Compare with neighbors: and , both greater than , confirming a minimum at .

Choosing the learning rate

The learning rate is the one knob you must set, and it matters. Consider descending from (slope ):

  • With : steps go — smooth convergence to .
  • With : , so the iterates oscillate forever — never converging.
  • With : the iterates grow — divergence.
  • With : correct direction, but thousands of steps are needed.

A practical rule of thumb: if the loss grows or oscillates, lower ; if progress is glacial, raise it. Modern training often shrinks over time — big steps early to move fast, small steps later to settle precisely. This tuning problem is a core part of the next lesson's topic, where gradient descent is applied to actual regression models.

Interactive plot of y = x² and the update line y = (1 − 2η)x through the origin, with a slider for the learning rate η from 0 to 1.2; the line's slope controls convergence, oscillation or divergence.-4-224510f(x) = x²update: x_new = (1 − 2η)x
One step of descent on x² from x = 3 follows the update line x_new = (1 − 2η)x. Drag η: below 0.5 the slope of this line is between −1 and 1 and iterates converge; at η = 0.5 it is flat; at η = 1 the slope is −1 and the walk bounces between 3 and −3 forever; beyond that it diverges.

Example 5. When the learning rate is too large Descend from with learning rate . What happens after one step?

The gradient at is .

The step jumped clear across the minimum at , landing at the mirror point . Repeating gives , so the iterate bounces between and forever and never reaches the minimum. The fix is a smaller learning rate.

Common mistakes

  • Adding the gradient instead of subtracting it, which climbs uphill and increases the loss — the update is .
  • Forgetting that the gradient must be re-evaluated at the new point after each step, not reused from the old point.
  • Assuming a zero gradient always means a minimum — it can also be a maximum or a saddle point, so check the second derivative or the loss value.
  • Using a learning rate so large the iterates oscillate or diverge, then concluding the minimum does not exist.
  • In multiple dimensions, updating only one coordinate — every coordinate must be updated with its own partial derivative simultaneously.

Practice

  1. Find the derivative of .
  2. For , evaluate the slope at .
  3. Find the critical point of , i.e. the value of where the derivative is zero.
  4. Minimize by gradient descent starting at with learning rate . What is the new point after one step?
  5. Continuing the same descent on from with learning rate , what is the point after two steps?
Answers
  1. , by differentiating each term with the power rule.
  2. , since .
  3. , from solving .
  4. , since the slope at is and .
  5. , since has slope and .

Next lesson: applying gradient descent to fit linear and logistic regression models, where the hill is the error between predictions and data.

Part of the free WordGPT Learn section. WordGPT is an AI writing workspace for documents, resumes, websites and study.

    Gradient descent — Math: from algebra to machine learning | WordGPT