The Multilayer Perceptron

From a Neuron to a Layer

A single neuron takes the whole input, weights it, sums, and squashes: $a = g(w \cdot x + b)$ . One number out.

Now line up several neurons, all reading the same inputs, but each with its own weights and bias. That is a layer.

If the layer has three neurons, you get three numbers:

$a_1 = g(w_1 \cdot x + b_1)$
$a_2 = g(w_2 \cdot x + b_2)$
$a_3 = g(w_3 \cdot x + b_3)$

Those outputs $[a_1, a_2, a_3]$ form a vector. So a layer transforms one vector into another, and that output vector can then become the input to yet another layer.

A layer is just neurons in parallel. It maps an input vector to an output vector, and that is the piece we stack.

The Anatomy of a Network

Chain a few layers together and you have a multilayer perceptron (MLP), the classic neural network:

Input layer: the raw features $x$ (not really neurons, just the numbers going in).
Hidden layers: one or more layers in the middle. This is where the network bends and re-represents the data.
Output layer: the final answer, whether a number, a probability, or a class score.

Data moves in one direction only, from input toward output, which is why an MLP is called feedforward. Three words worth pinning down:

Term	Meaning
Width	how many neurons sit in a layer
Depth	how many layers the network has
Parameters	all the weights and biases, the numbers training will tune

A Layer Is a Matrix Multiply

Writing $w_1 \cdot x$ , $w_2 \cdot x$ , $w_3 \cdot x$ one at a time is clumsy. So stack the neurons’ weight vectors as the rows of a matrix $W$ , and every dot product happens at once:

$z = Wx + b, \qquad a = g(z)$

Reading the shapes:

$W$ has one row per neuron and one column per input. A layer of $m$ neurons reading an $n$ -vector has a $W$ of size $m \times n$ .
$b$ is the bias vector, one entry per neuron.
$z = Wx + b$ gives all the weighted sums in a single multiply, and $a = g(z)$ bends each one.

One layer = one matrix multiply, one bias, one elementwise squash. No loops, no per-neuron bookkeeping.

The Forward Pass

Running an input through the whole network is called the forward pass, and it is now easy to say: just do matrix multiply, add bias, bend once per layer, feeding each result into the next.

For a three-layer network, the entire computation is:

\begin{aligned} a^{(1)} &= g(W^{(1)}x + b^{(1)}) \\ a^{(2)} &= g(W^{(2)}a^{(1)} + b^{(2)}) \\ \hat{y} &= W^{(3)}a^{(2)} + b^{(3)} \end{aligned}

That is all a neural network does to turn an input into a prediction. Two consequences worth holding onto:

This is why GPUs run deep learning. The forward pass is a chain of matrix multiplications, and matrix multiplication is precisely what a GPU does fast and massively in parallel.
The $W$ s and $b$ s are all there is to learn. Everything a network “knows” lives in those numbers. Training is nothing but searching for good values of them.

We can now take any input and produce an output. But two big questions are still open: what can a network like this actually represent, and how does it find those good weights? We take on representation first.