Why One Line Is Not Enough

A Wall for a Single Neuron

The perceptron was elegant: one neuron, one straight line. For a huge range of problems, that is plenty. But in 1969, Minsky and Papert pointed out a tiny problem a single neuron cannot solve, and it was so damaging it nearly froze the whole field for a decade.

Here it is, side by side with one a neuron handles easily:

OR Is Easy

Start with OR: fire if either input is on.

Plot those four points. Only $(0,0)$ is a $0$, sitting alone in the corner. One straight line fences it off from the rest, so a single neuron learns OR without breaking a sweat. AND is the same story.

XOR Is the Wall

Now XOR, exclusive or: fire when exactly one input is on.

Plot these. The two 1s land on opposite corners. The two 0s land on the other two opposite corners. The classes are diagonally interleaved.

Now try to draw one straight line with both 1s on one side and both 0s on the other.

You can’t. Every line, at every angle, gets at least one point wrong. A single neuron is mathematically unable to learn XOR.

And this is not a quirk of a toy puzzle. Plenty of real patterns curve like this, and one straight cut will never capture them.

The Fix: Let Neurons Team Up

One neuron gives one line. So give the network two neurons in a hidden layer, and let a neuron on top combine their answers.

Watch what two lines manage that one cannot. Drop one line just below the pair of 1s and another just above them. The region between the lines is a diagonal band, and that band holds exactly the two 1s.

You can also read it as plain logic:

• Hidden neuron 1 learns OR: is at least one input on?
• Hidden neuron 2 learns AND: are both on?
• The output neuron computes OR and not AND, which is “at least one on, but not both”, which is precisely XOR.

Each piece is a straight line a single neuron can learn. Stack them, and the straight cuts combine into a shape no single cut could make.

What the Hidden Layer Really Does

Here is the deeper view, and it is the idea the rest of deep learning rests on.

The hidden layer does not merely add lines. It builds a new space. Every input point gets remapped to a fresh position based on what the hidden neurons compute. For XOR, the hidden layer sends both 1-points to the same spot, right beside each other and far from the 0s.

And in that new space, the problem is suddenly trivial: a single straight line separates them.

A network cracks hard problems by re-representing the data, again and again, until a plain straight cut is all that is left to do.

One Missing Ingredient

XOR is small, but it proves something enormous:

• A single neuron is a linear model. It can only ever draw straight boundaries.
• Stacking neurons into layers lets a network fold those straight pieces into curves, and capture patterns no line can.

This is why we build deep networks instead of wide, flat ones. Each layer reshapes the data a little more, until the final neuron’s job becomes easy.

But there is a catch we quietly skipped. For stacking to actually add power, every neuron needs a non-linear activation sitting in the middle of it. Leave it out, and a stack of linear steps collapses back into… one linear step, one line, right back at the wall.

So the next question is unavoidable: what should that activation be? That is exactly where we go next.