Simple Machine Learning
A field journal · v.01
← Back
FiledApril 15, 2026·4 min read·By The Editor

Bias vs Variance in Machine Learning

Every prediction error splits into two parts — being too simple, and being too sensitive. You cannot zero both, the fixes are opposites, and telling them apart takes ten seconds.

theoryintuition

Every prediction error a model makes breaks into two parts.

Bias is error from the model being too simple to represent the truth. Variance is error from the model being too sensitive to the particular data it happened to see. You cannot drive both to zero at once, and the trade between them is the single most useful idea in applied machine learning.

Three models, same nine points

Too stiff — high bias
Too stiff — high biasNine scattered points with a flat fit drawn through them.
Gets it wrong the same way every time. Wrong on the training data, wrong on new data, and retraining changes nothing.
About right
About rightNine scattered points with a line fit drawn through them.
Catches the trend, ignores the jitter. Slightly wrong on training data — which is the price of being right on new data.
Too flexible — high variance
Too flexible — high varianceNine scattered points with a wiggle fit drawn through them.
Passes through every point exactly. Perfect on training data. Move one point and the whole curve thrashes.

The middle one is wrong about every single point. It is also the only one you would trust on data it has not seen.

In plain English

Bias is being confidently wrong in a consistent way. Variance is being right about this data and wrong about the next batch. The flat line has high bias — it is wrong the same way forever. The wiggly curve has high variance — it is perfect here and will be nonsense an inch to the left.

Why you cannot have both

Bias comes from restricting your model. Variance comes from freeing it. They are the same dial turned in opposite directions.

  • Make the model simpler → it can no longer chase noise → variance falls, bias rises.
  • Make the model more flexible → it can fit finer structure, including the noise → bias falls, variance rises.

Total error is roughly the sum of the two, plus noise you can never remove:

expected error=bias2+variance+irreducible noise\text{expected error} = \text{bias}^2 + \text{variance} + \text{irreducible noise}
In plain English

That third term is the part of the world your features simply do not explain. No model, however clever, gets below it. Chasing it is how people end up overfitting.

The chart

Figure 2 · the only chart that matters here
Training error against test error as model complexity growsTraining error falls steadily toward zero as complexity increases. Test error falls, reaches a minimum, then climbs again. The gap between the two curves is variance; the height of the joint low point is bias.model complexity →errorthe bargainbest tradeunderfittinghigh biasoverfittinghigh variancethis gapis variancetraining errortest error
Training error falls forever — give a model enough freedom and it will memorise the answers. Test error, the one you actually care about, falls and then turns back up. Everything to the left of the low point is bias; the widening gap to the right of it is variance.

Training error falls forever. That is why training accuracy is not evidence of anything — a model with enough freedom will drive it to zero by memorising, and a memoriser scores 100% on the exam it has already seen.

Test error is the one that matters, and it is U-shaped. It falls while the model is still learning real structure, bottoms out, then climbs as the model starts learning noise.

Diagnosing a model in ten seconds

Compare the two numbers. The answer is always one of three things:

Train error low
Test error high
Variance. It memorised. Get more data, simplify the model, or regularize.
Train error high
Test error high
Bias. It is too simple to represent the truth. More data will not help. Use a bigger model or better features.
Train error high
Test error low
Check your code. This is not a real result — usually a leak, a shuffled label, or a test set that is easier than it should be.

This matters because the fixes are opposites. More training data fixes variance and does nothing for bias. A bigger model fixes bias and makes variance worse. Getting the diagnosis backwards means every change you make moves the wrong direction, which is the single most common way a project stalls.

Why this explains so much else

Almost every technique you will meet is a move in this trade:

  • L1 and L2 regularization — deliberately add bias by penalising large weights, to buy a larger reduction in variance.
  • More training data — reduces variance without touching bias, which is why data usually beats cleverness.
  • Bigger models — reduce bias, and need more data or more regularization to keep variance in check. This is the entire story of why modern models are both enormous and trained on enormous corpora.
  • Ensembles — average many high-variance models so their errors cancel. Variance falls, bias stays put.
  • A held-out test set — the only instrument that can measure variance at all. Without one you are flying blind, which is why it is non-negotiable.

The short version

  • Bias = too simple, wrong in a consistent way, more data will not save you.
  • Variance = too sensitive, perfect on what it has seen, unreliable on what it has not.
  • Training error always falls; test error is U-shaped. Trust only the second.
  • Diagnose with the two numbers before you change anything, because the fixes are opposites.
← Archive— end of essay —Front page →