When One Label Isn’t Enough
k-means gave every point a hard answer: you belong to cluster A, full stop. But real clusters overlap, and a point sitting right between two of them is being forced to lie.
A Gaussian Mixture Model (GMM) fixes that. Instead of one label, it gives every point a soft membership, a probability of belonging to each cluster:
- k-means: “you’re in cluster A.”
- GMM: “you’re 54% A, 46% B.”
That single change means GMM can express doubt, which is exactly what you want for points near a boundary.
The Model: a Mixture of Blobs
The idea in the name: assume the data was generated by k Gaussian blobs (bell curves, which in 2D look like ellipses). Each blob has:
- a centre (its mean),
- a shape (its covariance, so it can be stretched and tilted),
- a weight (how much of the data it owns).
GMM’s job is to find the blobs that best explain the points. Every point then gets its soft membership from how well each blob fits it.
Fitting It: the EM Dance
There’s a chicken-and-egg problem:
- If we knew which blob each point came from → we could compute the blobs.
- If we knew the blobs → we could assign the points.
We know neither. Expectation-Maximization (EM) breaks the loop by alternating, exactly like the k-means dance:
- E-step (Expectation): given the current blobs, compute each point’s probability of belonging to each blob (its “responsibility”). → the soft assign.
- M-step (Maximization): given those soft memberships, update every blob’s centre, shape, and weight with a weighted average. → the move and reshape.
Repeat until the blobs stop moving.
EM is literally the soft version of k-means: the E-step is a soft assign, the M-step is a weighted move. Each round only improves the fit, so it always settles down.
Ellipses, Not Circles
There’s a bonus. Because k-means measures plain distance to a centre, its clusters are effectively circles. GMM’s covariance lets its blobs be stretched and tilted ellipses:
When a cluster is long and diagonal, k-means grabs the wrong points (its round region can’t follow the shape), while GMM wraps a tilted ellipse right around it.
GMM vs k-means
| k-means | GMM | |
|---|---|---|
| Assignment | hard (one label) | soft (probabilities) |
| Cluster shape | circles | stretched, tilted ellipses |
| Reports uncertainty? | no | yes |
| The dance | assign → move | E-step → M-step |
| Speed | faster | slower (more to fit) |
Think of GMM as k-means that admits when it isn’t sure and can bend its clusters to fit.
Why EM Matters Beyond GMM
Here’s the part worth remembering: EM is a general algorithm, not a GMM trick. Any time a model has hidden variables you can’t observe directly (here: which blob made each point), EM’s “guess the hidden thing, then update the model, repeat” recipe applies.
You’ve now seen the same alternating-optimization idea three times, k-means, GMM, and (soon) far bigger models. It is one of the workhorses of machine learning.