Overfitting vs Underfitting
Overfitting is memorising the training data; underfitting is never learning it. Both fail on new data, the fixes are exact opposites, and one ten-second check tells you which one you are looking at.
Overfitting is learning the training data too well. Underfitting is not learning it well enough. Both produce a model that fails on new data, and the fixes are opposites — so the first job is always to work out which one you have.
The exam analogy
Nobody would call the middle student a good student. They would say they memorised the practice paper. That is precisely what an overfitted model has done.
Overfitting is memorising instead of understanding. The model has learned the noise in your training data — the accidents, the one-off quirks — as though those were the pattern. They will never recur, so the knowledge is worthless.
How to tell which you have
Compare training error with validation error. There are only three outcomes:
| Training | Validation | Diagnosis | What to do |
|---|---|---|---|
| High | High | Underfitting | Bigger model, better features, train longer |
| Low | High | Overfitting | More data, regularization, early stopping |
| Low | Low | Fine | Ship it |
This takes ten seconds and determines everything you do next. Getting it backwards is the most common way to waste a week: adding data to an underfitted model changes nothing, and adding capacity to an overfitted one makes it worse.
Why models overfit
Give a model enough freedom and memorising is genuinely the easiest way to reduce training error. A model with more parameters than training examples can, in principle, store the answers outright.
Training error falls forever. Validation error falls, bottoms out, then climbs — and the climb is overfitting arriving. That U-shape and the theory behind it are covered in bias vs variance: underfitting is high bias, overfitting is high variance.
The fixes
Notice the last row points the opposite way. Every fix for overfitting makes underfitting worse, and vice versa. This is the whole reason the diagnosis has to come first.
Practical order of operations
- Overfit deliberately, first. Take 100 examples and check the model can drive training error to near zero. If it cannot, you have a bug or too little capacity — no amount of regularization will help.
- Then train properly and watch both curves.
- When validation turns up, stop, and add regularization or data.
- Touch the test set once, at the end. Tuning against it makes it useless as an estimate.
The traps
- Leakage. If information from the test set reached training — a duplicated row, a feature computed over the whole dataset — you will see suspiciously good results and a model that fails in production. Suspect this whenever results look too good.
- A validation set that is too small. Noisy validation numbers lead to confident wrong decisions.
- Tuning on the test set. Every decision made by looking at test performance fits your model to it a little more.
- Time-series data split randomly. If future rows land in training, the model is reading tomorrow's newspaper. Split by time.
The short version
- Underfitting: bad at training data and new data. Too simple.
- Overfitting: great at training data, bad at new data. Memorised.
- Diagnose by comparing training and validation error. Ten seconds, and it decides everything.
- The fixes are opposites. Never start changing things before diagnosing.
- Prove you can overfit a small sample first; it tells you capacity is not the problem.
Related
- Bias vs variance — the theory underneath.
- L1 vs L2 regularization — the standard fix.
- Epoch vs batch vs iteration — where early stopping happens.