The math of machine learning ยท Part 5 of 5
Neural networks and attention
How a neuron turns inputs into a prediction, how layers stack into a network, and how attention lets a model weigh what matters.
The math of machine learning11 min read
A spam filter, a photo tagger, and a chatbot all face the same problem: the answer depends on many inputs in complicated, nonlinear ways. A straight line through the data will not cut it. Neural networks solve this by chaining together very simple units, each doing nothing more than a weighted sum followed by a small squashing function, and stacking enough of them that simple pieces compose into flexible behavior.
Attention, the idea behind modern language models, tackles a second problem: not all inputs matter equally. In the sentence "The animal didn't cross the street because it was tired", the word "it" refers to "animal", and a model needs a way to learn that. Attention lets every input look at every other input and decide how much each one should influence its own representation, with the weights computed from the data itself.
A neuron: weighted sum, then activation
One artificial neuron takes an input vector , multiplies it by a weight vector , adds a bias , and passes the result through an activation function :
The weighted sum is linear, so on its own it could only ever draw straight boundaries. The activation is what bends the space. Common choices:
- ReLU: โ cheap, and the workhorse of hidden layers.
- Sigmoid: โ squashes any real number into , useful when the output is a probability.
- Softmax: turns a whole vector of scores into probabilities that sum to ; the -th entry is .
The weights are the parameters the network learns: training adjusts them so the network's outputs match the data. Mathematically, training is just the optimization you met earlier โ minimize a loss function over the weights, usually by gradient descent, with gradients computed by the chain rule (backpropagation).
Example 1. Computing one neuron by hand A neuron has weights , bias , and ReLU activation. What is its output for the input ?
First compute the weighted sum:
Then apply ReLU:
The output is . If had come out negative, ReLU would have clipped it to โ that clipping is exactly what makes the neuron nonlinear.
Stacking layers into a network
A layer is a whole set of neurons receiving the same input. If the input is a vector and the layer has neurons, the layer is a weight matrix , a bias vector , and the output is
You saw the matrix multiplication in the linear algebra lessons: row of holds the weights of neuron , so is that neuron's activation. A network stacks layers, feeding each output into the next:
The activation between layers must be nonlinear. Without it, two linear layers collapse into one: , just another linear map, and the whole network could only learn linear relationships. The nonlinearity is what buys the expressive power.
Between layers you can also insert other linear-algebra operations โ for example a PCA-like projection or a matrix multiply against another sequence, which is exactly what attention does below.
Example 2. A two-layer network forward pass A tiny network has one hidden layer with weights , bias , ReLU activation, and an output layer , , with sigmoid activation. Compute the output for .
Hidden layer pre-activation:
Apply ReLU (negative entries become ):
Output layer pre-activation:
Apply the sigmoid:
The output is about โ near , because is far from where the sigmoid sits at .
Why depth helps: composition of simple pieces
Each layer computes a new set of features from the previous layer's features. Early layers learn crude combinations of the raw inputs; later layers combine those into more abstract ones. Geometrically, each ReLU layer folds and re-stretches space a little, and a stack of such folds can carve up the input space far more finely than any single linear map.
This is the same composition idea you met with function composition and the chain rule: if layer is a function , the network is . Backpropagation is just the chain rule applied to that composition, computing how the loss changes with each weight so gradient descent can update it. Nothing more mysterious than that.
Example 3. Layers without ReLU collapse; nonlinear layers do not Show that composing the linear maps and is just another linear map, and contrast this with inserting a ReLU in between.
Compose them directly:
That is still of the form , an affine map (a linear map plus a constant). No matter how many layers without a nonlinearity you stack, the result is a single affine map, and with no bias terms it is a single linear map.
Now insert ReLU between them: , then , then :
This is a piecewise-linear function with a kink at โ something no single affine map can produce. Stacking many such kinked functions lets networks approximate very flexible curves; this is the content of the universal approximation idea.
Attention: queries, keys, and values
Suppose you have a sequence of input vectors โ say, one vector per word. A plain layer mixes them only through fixed weights, treating position 5 the same in every sentence. Attention instead computes, for each position, a data-dependent weighted average of all positions.
Each input is projected into three vectors:
the query, key, and value. Think of a search: the query is what position is looking for, each key is what each position advertises, and each value is the content it can contribute.
Position scores every position with a dot product, scaled down to keep softmax well-behaved:
where is the dimension of the keys. A softmax over the scores turns them into weights that are positive and sum to :
The new representation of position is the weighted average of the values:
That is the whole mechanism: dot products measure relevance, softmax normalizes relevance into weights, and the output is an average weighted by relevance. The projection matrices are learned, so the network learns what to attend to. In practice several of these run in parallel with different projections (multi-head attention), and each head can specialize in a different kind of relationship.
Example 4. Attention weights for three positions Suppose , , and for some query position the scores are , , . Compute the attention weights, then the output if , , .
Softmax over the scores:
Numerically, , so:
Check: . Good.
Now the weighted average of the values:
The output leans toward because position 1 had the highest score. Attention did not blend the three values equally โ it concentrated on the most relevant one, and how much it concentrates is controlled by the scores themselves.
Putting it together: a full attention output
The full pipeline for one position is: project to query, key, value; score the query against every key; softmax the scores; average the values with those weights. Notice which tools from earlier lessons do the work: dot products (linear algebra) measure similarity, softmax (exponentials and normalization) turns scores into probabilities, and weighted averages (expectation-like sums) produce the output. Training then adjusts and the layer weights by gradient descent on a loss, exactly as for any other parameter in the network.
A useful sanity check on any attention computation: the weights over must be strictly positive and sum to exactly , and the output must lie inside the convex hull of the value vectors โ attention can only mix values, never invent new ones. One caution: softmax weights are unchanged if you add a constant to all scores, but they do change if you multiply all scores by a constant โ scaling sharpens or flattens the distribution.
Example 5. A complete two-position attention computation Let (so ) with , , and projections , , (scalars). Compute the attention outputs for both positions.
Queries, keys, and values:
Scores for position 1:
Softmax weights:
Output for position 1:
Position 2 has different scores, because its query is larger: and . These are the position-1 scores multiplied by , and softmax is not invariant to scaling scores โ the larger scores sharpen the weights:
Output for position 2:
Both outputs lie between and , as attention outputs must, and both lean toward the larger value because position 2 has the larger key โ position 2 leans harder because its scores are larger and softmax concentrates them.
Common mistakes
- Forgetting the bias term and computing only ; the neuron computes , and the bias shifts the activation threshold.
- Applying an activation only sometimes or stacking purely linear layers, which collapses the whole stack into a single linear map; every hidden layer needs a nonlinearity.
- Softmaxing only the numerator's exponential and forgetting to divide by the sum of all exponentials; the weights must sum to across all positions.
- Assuming softmax weights stay the same when all scores are scaled by a constant; scaling scores sharpens or flattens the weights โ only adding a constant leaves them unchanged.
- Dividing by instead of in the scaled dot product; the standard scaling factor is , chosen to keep softmax out of its saturated regime.
Practice
- A neuron has weights , bias , and ReLU activation. Compute its output for the input .
- A sigmoid neuron computes with , , and . Compute its output, simplified to an expression in (you may use ).
- The scores entering a softmax over three positions are . Compute the third softmax weight and give it as a simplified expression in .
- Two vectors in dimensions are and . Using the scaled dot-product score , compute exactly in simplified radical form.
- For (so ), a position has query and attends over two positions with keys , and values , . Compute the attention output where is the softmax of the scores , giving the output as an exact expression in .
Answers
- : the pre-activation is , and ReLU clips negative values to .
- , so the output is .
- , since dividing top and bottom by removes one factor.
- , so .
- Scores are and , so , , and .
The next lesson uses these network building blocks to see how training dynamics and regularization shape what models actually learn.