Gradient Descent

Which Way Do We Nudge?

Last time we landed on one idea: learning is nudging a pile of numbers until the model stops being wrong.

But that leaves a giant hole. When the model guesses wrong, how does it know which way to nudge each number? Up or down? By a lot or a little?

Getting one number right by trial and error might be fine. But a real model has billions of them. You cannot guess. You need a method.

That method is gradient descent, and it is the single most important algorithm in all of machine learning.

First, a Number for “How Wrong”

Remember the “Mistakes so far” counter from the last note? Let’s make it precise.

At any moment, the model’s current numbers produce some total amount of wrongness. We squeeze all of it into a single number.

The loss is one number that measures how wrong the model is right now. High loss means very wrong. Zero loss means perfect.

Everything from here is about one goal: make the loss as small as possible.

A Landscape of Wrongness

Here is the picture that unlocks everything.

Imagine every possible setting of the model’s numbers spread out as positions on a landscape. At each position, the height of the ground is the loss for that setting. Bad settings sit high on the hills. Good settings sit low in the valleys.

Training is just finding the lowest point on this landscape. The bottom of a valley is where the model is least wrong.

But there’s a catch: you cannot see the landscape. It has billions of dimensions and nobody has a map. You start at a random spot (random starting numbers), blindfolded, in fog.

So you do the one thing you can do, feel the ground right under your feet. Even blindfolded, you can tell which way slopes downhill. So you repeat a tiny loop:

1. Feel which direction goes downhill from where you stand.
2. Take a small step that way.
3. Feel again from the new spot.
4. Repeat until the ground goes flat.

That’s the entire algorithm. Feel the slope, step downhill, repeat.

The Real Names

Every piece of that story has a technical name, and none of them are scary:

There is exactly one twist worth remembering. The gradient actually points in the direction of steepest increase, straight uphill. But we want to go down. So we always step in the opposite direction of the gradient.

That “step against the gradient” is literally why it is called gradient descent.

The One Knob That Ruins Everything

Of all the settings in machine learning, the learning rate (your step size) is the one people most often get wrong. Watch what the same valley does under three different step sizes:

• Steps too small → you will reach the bottom eventually, but it takes forever. Painfully slow.
• Steps too big → you leap clean over the valley, land on the far hillside, and overshoot back again. You bounce around and never settle. It can even get worse every step.
• Steps just right → you glide smoothly down into the valley.

Picking a good learning rate is one of the most important practical skills in the entire field. Too timid and you waste months. Too bold and the whole thing explodes.

One More Catch: You Might Get Stuck

Downhill has an honest limitation. It only promises to find a valley, not the deepest one.

If you start on the wrong slope, you roll into the nearest dip and stop, even when a much deeper valley sits right next door. A shallow valley is called a local minimum. The true deepest point is the global minimum.

In practice, this matters far less than you’d fear. In the billion-dimensional landscapes of real models, there’s almost always some downhill direction to keep going, so getting truly, hopelessly stuck is rare. But the honest truth stands: gradient descent guarantees a bottom, not the best bottom.

Why This Is the Whole Ballgame

Here is the payoff, and it is bigger than it looks.

This single algorithm, feel the slope and step downhill, is how essentially every model is trained. The cat-or-dog classifier from the last note, and a frontier model like the ones powering today’s AI assistants, are the same loop. The only difference is the size of the landscape: two dimensions for our toy, a trillion for the frontier model.

When you hear that a model was “trained for three months on ten thousand GPUs,” this is what that means. Three months of taking tiny downhill steps.

We now have both halves of learning:

• What to change → the pile of numbers (from note 1)
• Which way to change it → downhill on the loss (gradient descent)

That’s the engine. Everything ahead, every architecture, every frontier trick, is built on top of this exact loop.

def loss(guess):
    secret = 3
    gap = guess - secret
    return gap ** 2

print(loss(0))    # 9      way too low
print(loss(6))    # 9      way too high, equally wrong
print(loss(1.5))  # 2.25   getting warmer
print(loss(3))    # 0      nailed it

def slope(guess):
    poke = 0.0001

    wrong_on_the_left = loss(guess - poke)
    wrong_on_the_right = loss(guess + poke)

    change = wrong_on_the_right - wrong_on_the_left
    distance = 2 * poke
    return change / distance

print(slope(5))    # about  4.0   positive, so downhill is to the LEFT
print(slope(1))    # about -4.0   negative, so downhill is to the RIGHT
print(slope(3))    # about  0.0   flat, this is the bottom

guess = 0.0
step_size = 0.1

for i in range(30):
    slope_here = slope(guess)
    guess = guess - (step_size * slope_here)
    print(f"step {i}:  guess = {guess:.4f}   wrongness = {loss(guess):.4f}")

step 0:   guess = 0.6000   wrongness = 5.7600
step 1:   guess = 1.0800   wrongness = 3.6864
...
step 29:  guess = 2.9963   wrongness = 0.0000

def loss(guess):
    two_valleys = 0.1 * (guess * guess - 4) ** 2
    tilt = 0.3 * guess
    return two_valleys + tilt + 2

start guess = -3   ->   settles near -2.1   (the deep valley)
start guess =  3   ->   settles near  1.9   (a shallower one: a local minimum)