Fitting a Line

A New Kind of Question

Our first example was cat-or-dog: pick a category. But a huge share of real problems ask for a number instead:

• How much will this house sell for?
• How many views will this video get?
• What temperature will it be tomorrow?

Predicting a number is called regression. Let’s use a friendly example the whole way through: predict a house’s price from its size.

Look at the Data First

Imagine you collect real sales. Each house becomes a dot: its size along the bottom, its price up the side.

The dots don’t line up perfectly, but the trend is obvious, bigger houses cost more. The cloud slopes upward.

Your job is to draw a line through that cloud. Once you have it, you can predict the price of any house, even ones you’ve never seen.

The Model Is Just a Line

A straight line is described by two numbers:

$\text{price} = w \times \text{size} + b$

• $w$ is the slope: how much the price climbs for each extra unit of size. We call it a weight.
• $b$ is the intercept: the base price where the line starts. We call it a bias.

That’s the entire model. Those two numbers, $w$ and $b$, are the “pile of numbers” from note 1. A house-price model is literally just a good $w$ and a good $b$.

And making a prediction is dead simple: take a size, multiply by $w$, add $b$.

On the graph, a prediction is just: go up from the size, hit the line, read across to the price.

Which Line Is “Best”?

Here’s the problem. There are infinitely many lines you could draw. Which one is right?

Intuitively, the best line passes closest to all the dots. So we measure the miss. Watch how:

Three steps turn “how wrong is this line?” into a single number:

1. For each dot, take the gap between the real price and the line’s prediction. That gap is the error (or residual).
2. Square each gap.
3. Average all the squared gaps.

That final number is the Mean Squared Error (MSE).

Why square the gaps? Two good reasons:

• It makes every error positive, so misses above and below the line can’t cancel out.
• It punishes big misses much harder than small ones, so the line really tries to avoid being wildly off.

Low MSE means the line hugs the data. High MSE means it’s way off. MSE is the loss for linear regression.

The Payoff

Now look at what we’re holding:

• a model with parameters $(w, b)$, and
• a loss (MSE) that scores how wrong any choice of $w$ and $b$ is.

We already know exactly what to do with that. It’s a loss we want to shrink, so we reach for the tool from the last note: gradient descent. Start with a random junk line, feel which way to nudge $w$ and $b$ to lower the MSE, and step downhill. The line swings and slides into place until it hugs the data.

This is the first real model. Its parameters are $w$ and $b$, its loss is MSE, and it is trained by gradient descent. The three ingredients from note 1, now made concrete.

And a lovely bonus: the MSE landscape for linear regression is a perfect single bowl, one global minimum with no shallow traps. So here, gradient descent is guaranteed to find the one best line. No getting stuck.

One Quick Extension

Real houses have more than size: bedrooms, age, location. No problem. Just add more weights:

$\text{price} = w_1 \cdot \text{size} + w_2 \cdot \text{bedrooms} + w_3 \cdot \text{age} + \ldots + b$

It is still just a weighted sum plus a bias. And that shape, a weighted sum of inputs, is the single most important pattern in all of machine learning.

Hold on to it. A neuron, which we meet soon, is exactly this line, with one small twist on top.

sizes  = [1, 2, 3, 4, 5]
prices = [3, 5, 6, 8, 9]

def predict(size, w, b):
    return w * size + b

def mse(w, b):
    total = 0
    for i in range(len(sizes)):
        prediction = predict(sizes[i], w, b)
        gap = prices[i] - prediction
        total = total + (gap ** 2)
    return total / len(sizes)

print(mse(0, 0))        # 43.0    a flat line at zero, terrible
print(mse(1.5, 1.7))    # 0.06    the best-fit line, tiny error

def slope_for_w(w, b):
    poke = 0.0001
    higher = mse(w + poke, b)
    lower = mse(w - poke, b)
    change = higher - lower
    distance = 2 * poke
    return change / distance

def slope_for_b(w, b):
    poke = 0.0001
    higher = mse(w, b + poke)
    lower = mse(w, b - poke)
    change = higher - lower
    distance = 2 * poke
    return change / distance

w = 0.0
b = 0.0
step_size = 0.03

for i in range(400):
    slope_w = slope_for_w(w, b)
    slope_b = slope_for_b(w, b)
    w = w - (step_size * slope_w)
    b = b - (step_size * slope_b)

print(f"w = {w:.2f}   b = {b:.2f}   mse = {mse(w, b):.2f}")

w = 1.51   b = 1.68   mse = 0.06