The math of machine learning ยท Part 2 of 5
Linear and logistic regression
How linear and logistic regression turn prediction into a loss function that gradient descent can minimize.
The math of machine learning10 min read
Suppose you have data: hours studied and exam scores, square footage and price, pixels and "cat or not cat." You want a rule that predicts the answer for new inputs. Regression is the workhorse for this: pick a simple model, measure how wrong it is on the data, and adjust the model until the wrongness is as small as possible.
The previous lesson gave you the tool for the last step โ gradient descent, which walks downhill on a loss function. This lesson builds the two most important loss functions in machine learning: squared error for predicting numbers (linear regression) and log loss for predicting probabilities (logistic regression). Once you see how both are built, gradient descent trains them with the exact machinery you already have.
Linear regression: a line plus squared error
Linear regression predicts a number from an input using a straight line:
Here is the intercept and is the slope. Given data points , we measure how wrong the line is with the mean squared error (MSE):
Each term is a residual: the vertical gap between the line and a data point. Squaring does two useful things: it makes errors in either direction count positively, and it penalizes large errors much more than small ones, so the fit avoids systematically missing any point badly.
Training means choosing and to minimize . Two routes lead to the same place: solve the equations and directly (the normal equations), or run gradient descent. For a small dataset the normal equations are quick; for huge datasets gradient descent scales better, which is why it dominates in practice.
Example 1. Fitting a line by the normal equations Fit to the points , , by minimizing the squared error.
Compute the sums that appear in the normal equations:
Setting the partial derivatives of to zero gives:
Subtract the first equation from the second:
Substitute back:
Check by substituting into both normal equations: and . Both hold. The fitted line is , which passes exactly through all three points, so the minimum error is .
Training the line with gradient descent
The gradient of the MSE tells you how to nudge and . For a single data point with model , the loss is and the partial derivatives are:
Both derivatives share the factor โ the residual. Intuition: if the line predicts too high, the residual is positive, and the update pushes and down; if it predicts too low, they get pushed up. The update rule from the previous lesson applies directly:
where is the learning rate. With many points you either sum the gradients over all points (batch gradient descent) or use one point at a time (stochastic gradient descent) โ the loss landscape is a bowl either way, so descent converges to the same best fit.
Example 2. Two gradient descent steps on one data point Model fitted to the single point , so the loss is . Starting at with learning rate , compute two updates.
The derivative is:
Step 1, from :
Step 2, from :
Each step moves toward , cutting the distance to the minimum by a factor of per step (since ). Running many steps gives , the least-squares slope for a line through the origin fitted to .
Logistic regression: from a line to a probability
Many prediction problems have yes/no answers: spam or not, click or no click, sick or healthy. A line outputs any real number, but a probability must lie in . Logistic regression fixes this by squashing the line through the sigmoid function:
The sigmoid maps large positive near , large negative near , and to exactly . So the model is:
and is read as the probability that the answer is "yes." To make a hard decision, pick a threshold, usually : predict "yes" when , which happens exactly when since is increasing. The decision boundary is therefore the line .
Example 3. Evaluating the sigmoid and finding a decision boundary For the model , compute the predicted probability at , and find the value where the model switches its prediction.
At the linear part is:
Apply the sigmoid:
So the model says about a chance of "yes."
The prediction switches where , i.e. where :
Check: at , and , exactly on the boundary. For the model predicts "yes," for it predicts "no."
Why logistic regression does not use squared error
You might try to fit with squared error, but there is a better choice. The log loss (also called binary cross-entropy) for one labeled point with is:
Read it as two cases. If , the loss is : it is near when is near and blows up as . If , the loss is : it punishes confident wrong answers the same way. Log loss is the maximum-likelihood loss for a probabilistic model, it is convex in here (so gradient descent finds the global minimum), and its gradients are well-behaved even when the model is very wrong โ squared error's gradients vanish when the sigmoid saturates, which stalls training.
For one point, the gradient of the log loss with respect to the linear part has a beautifully simple form:
That is, the gradient is just prediction minus truth โ the same residual idea as in linear regression. This is not a coincidence: both models are members of the same family (generalized linear models), and it is why the same training loop trains both.
Example 4. Log loss for one example A model outputs for an example whose true label is . Compute the log loss. Then compare it with the loss if the model had output .
With , the loss is :
If the model had output for the same label:
The confident wrong answer costs about times more. Also check the gradient formula: with and ,
The negative gradient says should be increased, which makes sense: the model under-confidently predicted "yes" for a true "yes."
The shared training recipe
Strip away the details and both models train identically:
- Choose a model: for numbers, for probabilities.
- Choose a loss matched to the output type: squared error for numbers, log loss for probabilities.
- Compute the gradient of the loss with respect to the parameters.
- Update each parameter in the negative gradient direction with learning rate .
- Repeat until the loss stops improving.
The only difference between the two is the model's final squashing function and the loss. In deep learning this pattern generalizes: a network is a linear model followed by nonlinear transformations, the output layer picks an activation (none for regression, sigmoid or softmax for classification), and the loss is matched to that choice. The next lesson, backpropagation, is the algorithm that computes step 3 efficiently when the model has millions of parameters instead of two.
Example 5. One full logistic update Model with current , trained on one example with label . Using learning rate and the gradient , perform one update.
At :
The gradient is:
The update:
Check: at , , so the model moved toward predicting "yes," which is correct since the label is . Repeating this update drives (perfect confidence on this single example) โ with real datasets containing both labels, the two forces balance and settles at a sensible value.
Common mistakes
- Applying the sigmoid and then treating its output as a class label โ the sigmoid gives a probability; only compare it to a threshold (usually 0.5) to get a label.
- Training logistic regression with squared error โ use log loss instead, because squared error has vanishing gradients when the sigmoid saturates and is not the maximum-likelihood loss.
- Adding the gradient instead of subtracting it in the update โ the rule is , so a positive gradient must decrease the parameter.
- Forgetting the factor of in the slope derivative โ for the loss the derivative with respect to is , not .
- Assuming the fitted regression line must pass through the data points โ it generally does not; it minimizes the total squared vertical distance to them.
Practice
- Fit to the points , , by least squares. Set up and solve the normal equations for and .
- A linear regression model is . What value does it predict for ?
- Compute the sigmoid value .
- A logistic regression model computes before applying the sigmoid. For what value of is the model exactly on its decision boundary (predicted probability )?
- For linear regression on the single point with model , the squared-error loss is . Compute the partial derivative .
Answers
- , . The normal equations and come from the sums , , , .
- , since .
- , because so the denominator is .
- , because the boundary is where , i.e. .
- , by the chain rule applied to .
Backpropagation takes the gradient computation you did by hand here for two parameters and automates it for networks with millions of them.