Activation Functions

The Missing Piece

We ended the last note with a warning: stacking layers only helps if there is a non-linearity between them. Let us see exactly why.

A layer with no activation just computes a weighted sum, $z = Wx + b$ , which is a linear map. Stack two of them, and the algebra collapses:

$W_2(W_1 x + b_1) + b_2 = (W_2 W_1)\,x + (W_2 b_1 + b_2)$

The right side is still just $W'x + b'$ : one linear map. So a hundred stacked linear layers fold right back into a single line. Depth, on its own, buys nothing.

The escape is a tiny non-linear function, the activation, applied at every neuron. One bend, and the layers stop collapsing.

Without an activation, a deep network is a straight line. The activation is the kink that lets depth build curves.

What an Activation Actually Is

An activation $g$ is applied to the weighted sum, elementwise, right after it:

$a = g(z) = g(Wx + b)$

That is the entire change. Every neuron now does weighted sum, then bend. Those bends are what let a network carve curved boundaries instead of straight ones.

The only real requirements:

It must be non-linear, otherwise, as we just saw, nothing changes.
It should be differentiable, so gradients can flow back through it while training.

Meet the Family

Over the years, a handful of activations have dominated. Watch them stack up on the same axes:

Each is simply a different way to bend $z$ :

Activation	Formula	Range	In one line
Sigmoid	$\sigma(z) = \frac{1}{1+e^{-z}}$	$(0, 1)$	the original, squashes to a probability
Tanh	$\tanh(z)$	$(-1, 1)$	sigmoid, but centered on zero
ReLU	$\max(0, z)$	$[0, \infty)$	cheap, fast, the modern default
Leaky ReLU	$\max(0.01z,\, z)$	$(-\infty, \infty)$	ReLU with a small leak below zero
GELU	$z \cdot \Phi(z)$	$\approx(-0.17, \infty)$	smooth ReLU, used in transformers

(Here $\Phi$ is the standard normal CDF, so GELU weights each input by how likely it is to be “on”.)

The Real Question: the Gradient

Here is the part that actually separates a good activation from a bad one, and it has almost nothing to do with the output shape. A network learns by pushing gradients backward, so what truly matters is the slope of the activation.

Slide a point along each curve and watch its slope:

Sigmoid and tanh go flat at both ends. Out in the tails, the slope is almost zero. Stack many layers and those near-zero slopes multiply together into essentially nothing, so the earliest layers get no learning signal at all. This is the infamous vanishing gradient problem, and it made deep sigmoid networks nearly impossible to train.
ReLU has a slope of exactly 1 for every positive input. It never saturates on that side. That single fact is most of the reason deep learning finally began to work.

The output range is a distraction. An activation lives or dies by its gradient, and ReLU won because its gradient does not vanish.

ReLU’s one weakness: for negative inputs the slope is 0, so a neuron can get stuck outputting zero forever, a dead neuron. Leaky ReLU patches this with a small negative slope, and GELU’s smooth curve dodges the hard corner entirely, which is part of why transformers prefer it.

Which One Should You Reach For?

A short, practical map:

Hidden layers: default to ReLU. If too many neurons die, switch to Leaky ReLU. Inside transformers, use GELU.
Output layer: here you want saturation, because you want a bounded answer. Use sigmoid for a single probability, or its cousin softmax to pick one class out of many.
Sigmoid and tanh in hidden layers: mostly historical now. Their vanishing gradients make them a poor fit for depth.

We finally have every piece of a single neuron: a weighted sum, a bias, and a non-linear activation. The next move is to wire many of them together into layers, and see what a full network can represent.