The math of machine learning ยท Part 4 of 5
Principal component analysis
Principal component analysis: how to find the directions of greatest variance in data and use them to compress it.
The math of machine learning10 min read
Suppose you have a table of data with many columns โ height, weight, age, income, dozens of measurements per person. Much of it is redundant: columns that rise and fall together carry nearly the same information. If you could replace the many columns with a few new ones that capture most of the variation, you would have smaller data, faster algorithms, and often clearer pictures.
Principal component analysis (PCA) does exactly this. It rotates the coordinate system so that the new axes line up with the directions in which the data spreads out the most. The first axis (the first principal component) points along the greatest spread, the second along the greatest remaining spread at right angles to the first, and so on. Keeping only the first few axes is the compression.
The machinery for this is linear algebra you already have: the covariance matrix measures how the columns vary together, and its eigenvectors and eigenvalues give the directions and amounts of spread.
Centering the data
PCA measures spread, and spread is measured relative to the mean. So the first step is always to subtract the mean of each column from that column. After centering, the data cloud sits with its center of mass at the origin, and every direction through the origin is treated fairly.
Concretely, if the data points are rows of a matrix with rows and columns, compute the mean of each column and subtract it from every entry in that column. All later steps use the centered matrix.
Centering also fixes the interpretation of the first principal component: it is a direction through the center of the data, which is usually what you want when you describe how the data varies.
Example 1. Centering a small dataset Center the data points , , by subtracting column means.
Mean of the first column:
Mean of the second column:
Subtract from each point:
Check: the mean of each centered column is and , as required.
The covariance matrix
For centered data, the covariance matrix is a square matrix whose entry in row , column measures how column and column vary together:
where is the centered data matrix. The diagonal entries are the variances of the individual columns; the off-diagonal entries are covariances, telling you whether two columns tend to move together (positive), in opposition (negative), or independently (near zero).
is always symmetric, and symmetric matrices have two properties that make PCA work: all their eigenvalues are real, and their eigenvectors for distinct eigenvalues are orthogonal. Those eigenvectors will be our new axes.
Example 2. Computing a covariance matrix Compute the covariance matrix of the centered points , , using divisor .
Form by summing outer products of the centered rows.
First column times itself: .
Second column times itself: .
First column times second: .
So
Divide by :
The positive off-diagonal entry says the two columns tend to rise together.
Eigenvalues and eigenvectors: the principal components
The principal components are the eigenvectors of the covariance matrix , and the amount of variance along each is the corresponding eigenvalue. If is a unit eigenvector with eigenvalue , then the variance of the data projected onto the line spanned by is exactly .
Why? Projecting the data onto a unit vector gives new coordinates , and the variance of those coordinates is . For a unit eigenvector, , so . The eigenvalue is literally the variance captured by that direction.
The procedure for PCA is therefore:
- Center the data.
- Compute .
- Find the eigenvalues and eigenvectors of .
- Sort eigenvectors by decreasing eigenvalue; these are the principal components.
- To reduce dimension, keep the top eigenvectors and project: each data point becomes , where holds the kept eigenvectors as columns.
The sum of all eigenvalues equals the total variance of the original data, so the fraction tells you what share of the information a single component keeps.
Example 3. Finding the principal components of a 2x2 covariance matrix Find the eigenvalues and eigenvectors of and the variance fraction captured by the first component.
Solve :
So , giving :
For : gives , so ; a unit eigenvector is .
For : gives , so ; a unit eigenvector is .
The two eigenvectors are orthogonal, as expected for a symmetric matrix. The first component captures
of the total variance, so projecting onto the line keeps 75% of the spread.
Projecting data onto the components
Once you have chosen unit eigenvectors , reducing a data point is a simple dot product. A centered point becomes the -dimensional vector of scores
Each score is the coordinate of the point along that component. To reconstruct an approximation of the original point, multiply the scores back by the eigenvectors and add the mean:
Here is the centered point (the mean already subtracted), so adding puts the point back where it came from.
The reconstruction is exact if you keep all components (when equals the number of dimensions), and loses only variance in the discarded directions if you keep fewer. Because the eigenvectors are orthogonal, the scores are uncorrelated โ PCA produces new columns that do not repeat each other, which is one reason it is popular as a preprocessing step before fitting models.
Example 4. Projecting a point onto the first component Project the centered point onto the unit vector , and give the reconstructed point using only that component.
The score is the dot product:
The reconstruction from this single component is the score times the eigenvector:
Check: lies on the line , as any single-component reconstruction of this direction must, and the discarded part is orthogonal to since .
Choosing how many components to keep
The eigenvalues, sorted from largest to smallest, form the spectrum of the data. A common rule is to keep enough components to reach a target share of total variance, often 90% or 95%. Compute the cumulative sums divided by the total, and stop at the first index that reaches the target.
Another guide is the shape of the scree plot: eigenvalues plotted in decreasing order usually drop steeply and then flatten into a shallow tail; the components in the flat tail describe noise-like variation and are safe to drop.
Two practical cautions. First, PCA is sensitive to scale: if one column is measured in millimeters and another in meters, the first will dominate the covariance matrix purely because of units, so columns are often standardized (divided by their standard deviations) before PCA. Second, PCA is unsupervised: it finds directions of variance, not directions that best separate classes or predict a target. It reduces dimensions well, but a low-variance direction can still matter for prediction.
Example 5. How many components for 90% of the variance? A covariance matrix has eigenvalues (in decreasing order). How many components are needed to keep at least 90% of the total variance?
Total variance:
One component keeps
Two components keep
So two components suffice; the third adds only .
Common mistakes
- Forgetting to center the data before forming the covariance matrix, which makes the first component point toward the mean instead of along the spread โ always subtract column means first.
- Reporting eigenvectors as principal components without normalizing them to unit length, so the projected variance no longer equals the eigenvalue.
- Confusing eigenvalues with eigenvectors: the eigenvalue is the variance along the direction, the eigenvector is the direction itself.
- Applying PCA to raw columns with wildly different units or scales without standardizing, letting one large-scale column dominate all the components.
- Assuming the components with the smallest variance are useless for prediction โ PCA ignores any target variable, so low-variance directions can still carry class information.
Practice
- Center the data points , , by subtracting the mean of each column. What is the first coordinate of the second centered point?
- For a dataset whose covariance matrix is , find the two eigenvalues of , listed in decreasing order.
- The covariance matrix has eigenvectors along the coordinate axes. What fraction of the total variance lies along the first coordinate axis?
- Project the centered point onto the unit vector . What is the score (the dot product)?
- A dataset has covariance matrix . The first principal component is the unit eigenvector for the largest eigenvalue, taken with positive entries. Find the ratio of the second coordinate to the first coordinate of this unit eigenvector.
Answers
- 0, because the column mean is 3 and .
- 6 and 2, from solving .
- 5/6, since the total variance is and the first axis carries 5 of it.
- , since .
- . The largest eigenvalue is (the eigenvalues are ). For the second row of reads , so and .
PCA compresses data by finding low-dimensional structure by hand; in the next lesson, Neural networks and attention, you will see networks that learn such structure automatically.