Overfitting and Underfitting

The Real Test of Learning

Here is a question that changes everything: what are we actually trying to do?

In the last note, our line fit the house data nicely. But think about it, we already know the prices of the houses we trained on. Predicting those is pointless. The whole reason we built the model is to price houses we have never seen.

The goal was never to do well on the training data. It is to do well on new data. That is called generalization, and it is the only thing that actually matters.

Three Students

Picture three students prepping for an exam with a single practice test.

• The lazy one barely studied. Their rule is far too simple: “when in doubt, guess C.” They bomb the practice test and the real exam. This is underfitting (the model is too simple to capture the pattern).
• The crammer memorized the practice test’s answer key word for word, typos and all. They score 100% on the practice test, then fall apart on the real exam because the questions changed. This is overfitting (the model memorized the training data instead of learning the pattern).
• The good student actually understood the material. They do well on both. This is the sweet spot.

Overfitting is the sneaky one. It looks exactly like success, a perfect score, right up until it meets new data.

Where Complexity Comes From

How does a model “memorize”? It needs enough flexibility to do so.

Our straight line couldn’t memorize anything, it can only be straight. But if we let the model bend, by adding curves ($\text{size}^2$, $\text{size}^3$, and so on), it grows more and more flexible. Push that far enough and the curve can thread its way through every single training point exactly.

That flexibility is called model capacity, and it behaves like a dial:

Turning the dial gives three very different outcomes:

• Too little (a straight line when the data actually curves) → underfitting. It misses the real trend.
• Just right (a gentle curve) → it captures the pattern.
• Too much (a wild, wiggly curve) → overfitting. It nails every training point but swings crazily between them, treating random noise as if it were signal.

The Tell: Training Error vs Test Error

So how do you actually catch overfitting before it embarrasses you? Track two numbers as you turn up the dial.

• Training error just keeps falling. A wiggly enough curve can hit every training point, all the way down to zero.
• Test error traces a U-shape: it drops as you move from too-simple to good, then climbs back up as you start to overfit.

The bottom of that U is the model you want. Watching training and test error split apart is the core diagnostic in all of machine learning.

Seeing It Fail on New Data

Here is the whole point, side by side. Two models that both fit the training dots, tested on fresh points they’ve never seen:

The overfit model aced the training data. But on new points it lurches all over the place, because it learned the noise, not the trend. The simpler, smoother model, which never scored perfectly on training, quietly wins where it counts.

A perfect training score is not a trophy. It is often a warning.

The Names (We Go Deeper Next Time)

Two bits of vocabulary, worth seeding now:

• Underfitting is also called high bias (the model is too rigid, its built-in assumptions are wrong).
• Overfitting is also called high variance (the model is too twitchy, it swings wildly with the exact training points it happened to see).

Balancing these two is the bias-variance tradeoff, and it earns its own note next.

Why This Is Everywhere

This is not a beginner’s footnote. It is the central struggle of all machine learning, frontier models included. A model with billions of parameters can absolutely memorize its training data. The entire art is forcing it to generalize instead.

Nearly every technique you will meet later, gathering more data, regularization, dropout, exists for one reason: to fight overfitting. Keep this picture close. You will see it again at every scale, from a straight line to a trillion-parameter model.

train_cups, train_focus = [0, 0.7, 2.0, 2.6, 4.0, 4.6, 6.0], [1.8, 3.6, 6.2, 7.0, 6.5, 5.3, 1.5]
test_cups,  test_focus  = [1.3, 3.3, 5.3], [5.4, 6.8, 4.0]

def normalize(cup):
    return (cup - 3) / 3

def predict(cup, weights):
    scaled = normalize(cup)
    total = 0
    for power in range(len(weights)):
        total += weights[power] * scaled ** power
    return total

def training_loss(weights):
    total = 0
    for i in range(len(train_cups)):
        gap = train_focus[i] - predict(train_cups[i], weights)
        total += gap ** 2
    return total / len(train_cups)

def fit(degree, rounds=8000, step_size=0.05):
    weights = [0] * (degree + 1)
    poke = 1e-5
    for _ in range(rounds):
        slopes = []
        for knob in range(len(weights)):
            up, down = weights[:], weights[:]
            up[knob] += poke
            down[knob] -= poke
            slopes.append((training_loss(up) - training_loss(down)) / (2 * poke))
        for knob in range(len(weights)):
            weights[knob] -= step_size * slopes[knob]
    return weights

def memorizer(cup):
    total = 0
    for i in range(len(train_cups)):
        piece = train_focus[i]
        for j in range(len(train_cups)):
            if j != i:
                piece *= (cup - train_cups[j]) / (train_cups[i] - train_cups[j])
        total += piece
    return total

def error(model, cups_here, focus_here):
    total = 0
    for i in range(len(cups_here)):
        total += (focus_here[i] - model(cups_here[i])) ** 2
    return total / len(cups_here)

line, curve = fit(1), fit(3)

def straight_line(cup): return predict(cup, line)
def gentle_curve(cup):  return predict(cup, curve)

for name, model in [("line", straight_line), ("curve", gentle_curve), ("monster", memorizer)]:
    print(f"{name:8} train {error(model, train_cups, train_focus):.2f}   test {error(model, test_cups, test_focus):.2f}")

line     train 3.42   test 3.55
curve    train 0.18   test 0.29
monster  train 0.00   test 6.90