FiledSeptember 16, 2026·5 min read·By The Editor

SGD vs Momentum vs Adam

All three follow the slope downhill; they differ only in what they remember. SGD remembers nothing, momentum remembers direction, Adam remembers direction and scale — and one of them has a bug you should know about.

optimizationintuition

All three do the same thing: look at the slope, take a step downhill, repeat. They differ only in how much they let the previous steps influence the current one.

Plain SGD has no memory. Momentum remembers direction. Adam remembers direction and per-weight step size. That is the entire hierarchy.

The problem they are all solving

Figure 1 · the same valley, three ways down

Loss landscapes are rarely round bowls. In a long narrow ravine, the steepest direction points across the valley rather than along it — so plain SGD bounces wall to wall while barely advancing. Momentum and Adam are both answers to this one shape.

Loss landscapes are not round bowls — they are long, narrow ravines. In that shape, the steepest direction points across the valley rather than along it. Plain gradient descent follows the steepest direction faithfully, so it bounces from wall to wall, making very little progress toward the actual bottom.

SGD: no memory

w \leftarrow w - \eta \, \frac{\partial L}{\partial w}

Take the slope, multiply by the learning rate, subtract. This is the rule worked out by hand in gradient descent by hand, and everything below is a modification of it.

Its weakness is visible in the figure: each step is decided entirely by the local slope, so a ravine produces zig-zag. Its strength is that it is simple, cheap in memory, and — with a well-tuned learning rate and schedule — still produces the best final results on some vision tasks.

Momentum: remember the direction

Instead of stepping by the gradient, accumulate a running velocity and step by that.

v \leftarrow \beta v + \frac{\partial L}{\partial w} \qquad w \leftarrow w - \eta v

In plain English

β (usually 0.9) is how much of the old velocity you keep. Directions you keep going build up speed; directions that flip sign every step cancel themselves out. It is a heavy ball rolling downhill instead of a hiker recalculating at every footstep.

In the ravine, the across-valley components alternate and cancel, while the along-valley components agree and accumulate. The zig-zag smooths out and progress speeds up — for one extra number per weight.

Adam: remember direction and scale

Adam adds a second memory: how big this particular weight's gradients have recently been. Each weight then gets its own effective step size.

m \leftarrow \beta_1 m + (1-\beta_1) g \qquad v \leftarrow \beta_2 v + (1-\beta_2) g^2

w \leftarrow w - \eta \, \frac{\hat{m}}{\sqrt{\hat{v}} + \epsilon}

In plain English

m is momentum — the average recent gradient. v is the average recent gradient squared, which measures how jumpy this weight has been. Dividing by √v means consistently-large gradients get damped and consistently-tiny ones get amplified. Every weight ends up with its own learning rate, set automatically.

That last property is why Adam became the default. A big model has weights whose gradients differ by orders of magnitude, and no single global learning rate suits all of them. Adam handles that without being asked.

The catch: it keeps two extra numbers per weight, so optimiser state costs twice the model size in memory. On a large model that is a real constraint.

Which to use

Just starting

Adam, learning rate 3e-4. It works on almost anything without tuning, which is why it is everywhere.

Transformers / NLP

AdamW. Effectively the only choice in practice.

Image models, chasing the last 1%

SGD with momentum. Slower to tune, and it still tends to generalise slightly better.

Training is unstable

Lower the learning rate before you change optimiser. It is the learning rate nine times out of ten.

AdamW is worth knowing by name. Standard Adam interacts badly with L2 regularization — the penalty gets divided by that same √v term, so weights with large gradients end up barely regularized at all. AdamW applies the decay separately, as intended. If you are using Adam with weight decay, you want AdamW. This is not a subtle preference; the original is simply wrong here.

Things worth knowing

The learning rate still matters most. Adam adapts the relative step per weight, not the overall scale. A bad global learning rate ruins Adam exactly as it ruins SGD.
Schedules beat optimiser choice. Warmup then cosine decay is close to standard, and moving from a constant rate to a schedule usually buys more than switching optimisers.
Adam converges faster; SGD sometimes lands better. A real and slightly annoying finding: Adam gets there sooner, well-tuned SGD+momentum occasionally ends up at a solution that generalises better.
NaN is not an optimiser problem. It is almost always too high a learning rate, or exploding gradients. Lower the rate and clip the gradients.

The short version

All three are "follow the slope downhill." They differ in memory.
SGD: no memory, zig-zags in ravines, cheapest.
Momentum: remembers direction, cancels the zig-zag, one extra number per weight.
Adam: remembers direction and per-weight scale, needs the least tuning, two extra numbers per weight.
Use AdamW, not Adam, whenever weight decay is involved.
Default: Adam at 3e-4. Tune the learning rate before anything else.

Gradient descent, by hand — the base rule, in full arithmetic.
Epoch vs batch vs iteration — what one "step" actually covers.
Hyperparameters vs parameters — the learning rate is the one that matters most.
L1 vs L2 regularization — why AdamW had to exist.

← Archive— end of essay —Front page →