The Bold Claim
Forget the intimidating name for a moment. Here is what a function really is: you put a number in, you get a number out, and if you plot the two you get a shape, whether a line, a wiggle, or a hill.
So the real question of this note is simple:
What shapes can a neural network draw?
The answer is genuinely surprising. A network with just one hidden layer, given enough neurons, can draw any curve you want, as closely as you want. That is the universal approximation theorem (Cybenko 1989, Hornik 1991), and it is the formal reason people say “neural networks can learn anything.”
Let us build up why, one piece at a time.
One Neuron Is One Bend
A single ReLU neuron can only draw the most boring shape there is: flat for a while, then a straight ramp. One bend.
That is all a single neuron does. But notice its two dials:
- The bias slides the bend left or right.
- The weight sets how steep the ramp is, and flipping its sign points the hinge the other way.
One neuron, one adjustable bend. Hold onto that, because everything else is just adding bends together.
A Few Bends Make a Bump
Here is the first bit of magic. The output neuron adds up all the hidden neurons, so their bends stack. Put a couple of ramps together and you carve out a bump:
An up-ramp, then a down-ramp, then a flattening ramp, and you have built a little hill sitting at one spot. On its own that bump is nothing special. But a bump is a piece of a function, a single feature parked exactly where you want it.
Two or three bends give you one bump, and a bump is a tile you can lay down anywhere.
Enough Bends Trace Any Curve
Now the payoff. Lay down many bumps, each at its own place and its own height, and their sum traces out any curve, exactly the way a row of thin tiles fills in a shape.
Watch the fit close in on the target as neurons are added. 2 neurons gives a blocky sketch. 4 finds the rough shape. 16 is almost the exact curve. Want it closer still? Add more neurons.
This is the whole theorem: one hidden layer, enough neurons, any curve. Each neuron adds a bend, and enough bends make any shape.
The Fine Print
This result is famous, and famously oversold. It promises only that a network can draw the curve. It stays completely silent on everything else that matters:
- It does not say training will find it. “Some setting of the weights works” is a world apart from “gradient descent will actually land there.”
- It can demand a ridiculous number of neurons. For a genuinely wiggly function, a single hidden layer might need thousands of neurons to get close.
- It says nothing about new data. Nailing every training point is not the same as being right on examples you have never seen.
So “can represent” is a far weaker promise than it sounds. It describes what exists, not what you can train, or trust.
So Why Bother With Depth?
If one wide layer can already draw anything, why stack many layers at all? Because depth is dramatically more efficient.
Each new layer can fold the function and double its number of wiggles. So a deep stack reaches wiggles with just tiny layers, while a single wide layer would have to add one neuron per wiggle, thousands of them, to keep pace.
A shallow network can do it. A deep network does it with exponentially fewer neurons. Width is enough in principle, but depth is what makes it practical.
That settles what a network can represent. The last piece of the puzzle is the very thing universal approximation pointedly refused to answer: how does a network actually find the right weights? That is training, and it runs on an algorithm called backpropagation, which is exactly where we head next.