Principal Component Analysis

Squeezing the Columns

k-means grouped the rows of your data (the individual points). Now the opposite problem: too many columns (the features).

Real datasets can have hundreds of features, and that hurts in three ways:

• Slow: every extra column costs compute and memory.
• Unplottable: you can’t eyeball more than 2 or 3 dimensions at once.
• Distances break down: the curse of dimensionality makes everything look equally far apart.

So we want to shrink those features down to a handful that still capture what matters. That’s dimensionality reduction, and PCA (Principal Component Analysis) is the classic.

Data Is Flatter Than It Looks

Here’s the key realization: real data almost never fills all its dimensions, because features constantly repeat each other:

• height in cm and height in inches → literally the same number, twice.
• a car’s horsepower and its top speed → they rise and fall together.
• hours studied and test score → tightly linked.

Each pair is really one underlying thing wearing two hats.

Pile up enough of that redundancy and a dataset with 100 columns might genuinely live on a flat 5-dimensional sheet floating inside that space.

Almost all the “extra” dimensions are wasted. PCA’s job is to find that sheet and throw the waste away.

The Best Angle to Flatten It

Think of it like photographing a fish:

• from the side → you see the whole fish, every detail spread out.
• head-on → it collapses to a tiny blob, all the detail hidden behind itself.

Flattening data is exactly the same: the angle you choose decides how much you keep.

PCA rotates through every possible angle and keeps the one where the shadow spreads out the most, because spread is where the information lives. That gives you a ranked set of axes:

• PC1: the direction of maximum spread (the fish’s long axis). The most informative.
• PC2: the next-best direction, always perpendicular to PC1.
• PC3, PC4, …: each perpendicular to the rest, each holding a little less.

Spread = information. The direction things vary along is the direction worth keeping.

Keep the Long Axes, Drop the Short

Once you have the axes, compressing is easy: project every point onto the top few and forget the rest.

The tiny-spread axis (PC2 here) held almost nothing, so collapsing onto PC1 barely changes the layout. That single move buys you a lot:

• Smaller data: two columns became one here, or a hundred into five at scale.
• Denoising: the dropped directions were mostly noise.
• A picture: squash down to 2D and actually see your data.

How Many Do You Keep? Variance Explained

Each component reports what fraction of the total spread it captures. A typical breakdown:

Keep just enough components to cover the variance you care about, often ~95%. Here that’s the first three, everything after is redundancy and noise, safely dropped.

(Under the hood, the components are the eigenvectors of the data’s covariance matrix, found via eigendecomposition or SVD. But the whole idea is nothing more than “directions of maximum spread.“)

The Catch: PCA Only Cuts Straight

PCA draws straight axes, quietly assuming the real structure is flat. When the data actually curves, a straight projection is a disaster:

That spiral has a perfectly clear order (the colour gradient), yet flattening it onto one straight line folds distant parts of the curve on top of each other, and the order dissolves.

Curvy structure needs nonlinear methods instead. The famous pair, t-SNE and UMAP:

One caution about those pretty pictures: in a t-SNE or UMAP plot the distances and cluster sizes mean nothing. Read them only as “these things belong together.”

The Big Connection

Step back and notice what PCA really does: it turns bloated data into a few meaningful numbers. That is exactly what embeddings do in deep learning, squeezing a word or an image into a short vector that captures its meaning.

PCA is the classical ancestor of embeddings. Hold on to it, because when we reach neural networks, “compress the data into a few dimensions that carry the meaning” becomes one of the most important ideas in the whole field.