Gradient Descent, By Hand

Every explanation of gradient descent shows you a marble rolling down a smooth bowl, and then the article ends. The marble is doing physics. Gradient descent is doing arithmetic — six numbers and a subtraction, repeated.

So let us do the arithmetic. One knob, one data point, five steps, every symbol decoded. By the end of this page you will have run the algorithm that trains every neural network on earth, on paper, and you will know exactly what each piece of the notation is asking for.

The whole idea, in one sentence

You are standing on a hillside in thick fog. You cannot see the bottom. But you can feel which way the ground tilts under your boots — so you step that way, and then you do it again.

Figure 1 · the whole idea

That is it. That is gradient descent. Everything that follows — the squared loss, the derivative, the Greek letter — is bookkeeping for two questions:

1. Which way is downhill? (that's the derivative)
2. How big a step should I take? (that's the learning rate)

Answer those two, take the step, repeat.

Step 1 — We have exactly one knob

Our model is the simplest thing that can still be called a model: multiply the input by a number.

$$\hat{y} = w \cdot x$$

w is the knob. It is the only thing we are allowed to change. Training this model means finding the best number to put in that one slot.

Step 2 — One data point, one miss

Here is our entire dataset. One row:

$$x = 2, \qquad y = 5$$

Someone gave us an input of 2 and told us the right answer is 5. We start the knob at $w = 1$, because we have to start somewhere and 1 is a number.

$$\hat{y} = w \cdot x = 1 \cdot 2 = 2$$

We guessed 2. The truth was 5. We are off by 3.

Figure 2 · the miss

The line we are drawing is pinned at the origin, so the only thing $w$ controls is how steep it is. Turn $w$ up and the line swings up toward the point; turn it down and the line droops away. Somewhere in between, the line passes right through. Our job is to find that somewhere.

Step 3 — Wrongness is the miss, squared

We need a single number that says how badly we are doing. The obvious candidate — the miss, $y - \hat{y}$ — has a problem: missing high gives a negative number and missing low gives a positive one, so two mistakes could cancel out and look like success.

Fix: square it.

$$L(w) = (y - \hat{y})^2 = (y - wx)^2 = (5 - 2w)^2$$

Figure 3 · why we square it

At our starting knob setting $w = 1$:

$$L(1) = (5 - 2 \cdot 1)^2 = 3^2 = 9$$

Our wrongness is 9. We would like it to be less.

Step 4 — Every knob setting has a wrongness. Draw them all.

Nothing stops us from asking: what would the loss be at $w = 0$? At $w = 3$? At every value in between? Plot the answers and you get a curve.

Figure 4 · the hill, drawn out

This is the hill from Figure 1, drawn honestly. And here is the important part, the part that makes the fog metaphor accurate rather than decorative:

The model never gets to see this curve. Drawing it required us to know the answer already. All the algorithm ever gets is the ground under its feet: the loss where it is standing, and the tilt where it is standing.

Step 5 — The tilt is the derivative, and the notation is not scary

The tilt of the curve at the exact point where you stand has a name. It is the derivative, and it is written like this:

$$\frac{dL}{dw}$$

That fraction intimidates people, so let us take it apart completely.

Figure 5 · the derivative is just this slope

Geometrically it is dead simple: lay a straight stick on the curve so it touches at exactly one point. The steepness of that stick is the derivative. On our curve at $w = 1$, moving right by 0.5 drops you by 6, so the tilt is $-6 \div 0.5 = -12$.

Now, where does that number come from without drawing anything? Two rules, applied once each.

Our loss is something squared: $L = (5 - 2w)^2$. Give the inside its own name, $u = 5 - 2w$, so that $L = u^2$.

Rule one — the power rule. The derivative of $u^2$ is $2u$. (Squares grow at twice their side length. Widen a square by a hair and you add two new strips of area, one on each open edge.)

$$\frac{dL}{du} = 2u$$

Rule two — the chain rule. But $u$ itself moves when $w$ moves, so we need that rate too. $u = 5 - 2w$ shrinks by 2 for every 1 that $w$ grows:

$$\frac{du}{dw} = -2$$

The chain rule says these rates simply multiply — if the loss moves twice as fast as $u$, and $u$ moves twice as fast as $w$ in the other direction, then per unit of $w$ you get both effects at once:

$$\frac{dL}{dw} = \frac{dL}{du} \cdot \frac{du}{dw} = 2u \cdot (-2) = -4u = -4(5 - 2w)$$

Plug in our starting knob, $w = 1$:

$$\frac{dL}{dw}\Big|_{w=1} = -4(5 - 2 \cdot 1) = -4 \cdot 3 = -12$$

Same $-12$ we read off the picture. And the sign is the entire message: negative slope means the ground falls away to the right, so the bottom is to the right, so we should increase $w$.

Step 6 — The forward pass and the backward pass

We just got that $-12$ with algebra. Real frameworks do not use algebra. They use a computation graph, and once you see it, backpropagation stops being a mystery — it is the chain rule with somewhere to put the numbers.

Write our model out as a graph of tiny operations, one node per arithmetic step:

Figure 6 · the computation graph

The forward pass

Feed the numbers in on the left and let them flow right. Multiply $x$ by $w$ to get the prediction. Subtract that from $y$ to get the miss. Square the miss to get the loss. You come out the far end holding one number: $L = 9$.

The backward pass

Now walk the same graph backwards. Start at the loss carrying the number 1 — "how much does the loss change when the loss changes" — and every time you pass a node, multiply by that node's local slope.

passing backward through	the node's local slope	what you are now carrying
the loss itself	—	$1$
the squaring node	$\dfrac{dL}{du} = 2u = 6$	$6$
the subtract node	$\dfrac{du}{d\hat{y}} = -1$	$-6$
the multiply node	$\dfrac{d\hat{y}}{dw} = x = 2$	$\mathbf{-12}$

You arrive at the knob holding $-12$. Same number we read off the tangent line in Figure 5, same number the algebra gave us. Three routes, one answer — which is exactly the point.

Why frameworks do it this way

Three reasons, and all three are why loss.backward() exists at all.

Each node only needs to know its own slope. The squaring node knows one fact: "my slope is $2u$." It knows nothing about $w$, nothing about the loss, nothing about what comes next. That is why you can invent a new kind of layer, teach it one local rule, and drop it into a network that already works.

The forward pass has to be remembered. Look again at the squaring node's local slope, $2u$ — it needs the value of $u$ that came through on the way forward. Every node is like this. That is why training a model eats several times the memory of running one: the backward pass feeds on the forward pass's leftovers, so every intermediate value has to be kept alive until its gradient is taken.

One backward walk serves every knob at once. Our graph has a single knob, so the saving is invisible here. Give it a billion knobs and a backward pass still costs roughly what a forward pass costs — about the same work again, not a billion times more. The obvious alternative, nudging each knob a hair and re-running the model to see what happened, would cost a billion forward passes per step. That one fact is why deep learning is possible at all.

So the training loop, in the order it actually happens:

1. Forward — push a batch of data through the graph, land on a loss.
2. Backward — walk back through the same graph, collecting one slope per knob.
3. Step — apply $w \leftarrow w - \eta \frac{dL}{dw}$ to every knob, using the slopes you just collected.
4. Repeat — next batch, a few hundred thousand times.

Backpropagation is step 2, and only step 2. It computes slopes and never moves a single knob. Gradient descent is step 3: it moves the knobs and does no calculus. The two get said in the same breath so often that people assume they are one thing. They are two, and the graph is what hands the first one's answer to the second.

Step 7 — The update rule, symbol by symbol

Here is the rule that does the work. It is one line, and once decoded there is nothing left in it.

$$w_{\text{new}} = w_{\text{old}} - \eta \, \frac{dL}{dw}$$

The minus sign trips everybody, so state it plainly: the derivative points uphill. A positive slope means the loss rises as $w$ rises. We want the loss to fall, so we move against the slope — hence subtract. When the slope is negative, subtracting a negative moves us right. The sign handles itself; you never have to think about the direction again.

Step 8 — Do the arithmetic

We have every number we need. Set the learning rate to $\eta = 0.1$, which is a guess, because it always is.

$$w_{\text{new}} = 1 - 0.1 \cdot (-12) = 1 + 1.2 = 2.2$$

The knob moves from 1 to 2.2. Check the new wrongness:

$$L(2.2) = (5 - 2 \cdot 2.2)^2 = (5 - 4.4)^2 = 0.6^2 = 0.36$$

Figure 7 · the step, taken

Nine down to 0.36, in one step. Not because anything clever happened — because the wall was steep there and the bottom was close. Do not expect this from a real model.

Step 9 — Do it four more times

Same three moves, over and over: measure the tilt, multiply by 0.1, subtract. Click through the steps.

And the same five steps as a ledger:

step	knob w	prediction 2w	miss 5−2w	loss L	slope dL/dw	move −η·slope
0	1	2	3	9	−12	1.2
1	2.2	4.4	0.6	0.36	−2.4	0.24
2	2.44	4.88	0.12	0.0144	−0.48	0.048
3	2.488	4.976	0.024	0.000576	−0.096	0.0096
4	2.4976	4.9952	0.0048	2.304×10−5	−0.0192	0.00192
5	2.49952	4.99904	0.00096	9.216×10−7	−0.00384	0.000384

Two things are worth staring at in that table.

The steps shrink on their own. Nobody told them to. The step size is the slope times 0.1, and the slope flattens as you approach the bottom — so the algorithm brakes automatically as it arrives. The first step moved the knob by 1.2; the fifth moved it by 0.000384.

The miss shrinks by the same factor every time. 3, then 0.6, then 0.12, then 0.024 — each one is a fifth of the last. That factor is set entirely by $\eta$, which is the next section.

Step 10 — The learning rate is the whole ballgame

$\eta$ (eta) is the only number in this algorithm that you have to choose, and nothing tells you what it should be. Here is the same hill, the same rule, the same starting knob, with three different values.

For our particular toy problem the arithmetic is clean enough to state exactly. Each step multiplies the remaining miss by $(1 - 8\eta)$, which tells you everything:

• $\eta = 0.125$ — the miss is multiplied by zero. One step lands on the exact answer. (Nobody gets this lucky twice.)
• $\eta < 0.125$ — you undershoot every time and creep in. Safe.
• $0.125 < \eta < 0.25$ — you overshoot, land on the far wall, and bounce back in. Ugly but it converges.
• $\eta = 0.25$ — the miss is multiplied by $-1$. You ping-pong between two points forever.
• $\eta > 0.25$ — the miss grows every step. The model diverges into NaN and your training run is over.

Step 11 — What changes in a real model

Everything you just did is what actually runs inside a large model. Four things get bigger; none of them get different.

More knobs

One knob gave us a curve. Two knobs give a landscape, and the tilt becomes a pair of numbers — how steep along knob 1, how steep along knob 2. That list of slopes, one per knob, is the gradient, written $\nabla L$.

Figure 9 · more than one knob

A production language model has hundreds of billions of knobs, so the gradient is a list hundreds of billions of numbers long. The update rule does not change by a single character; it just runs on every entry at once.

Lumpier hills

Our bowl had exactly one bottom, which made downhill and correct the same direction. Real loss landscapes do not.

Figure 10 · real hills are lumpy

More than one data point

We had one row. Real training sets have millions, so the loss is the average over examples, and the gradient is the average of their gradients. Averaging over all of them for every step is too slow — so you average over a random handful (a minibatch) and accept a noisy estimate of the tilt. That is the "stochastic" in stochastic gradient descent. Noisy tilt, cheap step, far more steps per hour.

Fancier steps

Momentum, Adam, learning-rate schedules, weight decay — every one of them is a refinement of the same two questions from the top of this page. Momentum keeps a bit of your last step so you coast through flat patches. Adam gives every knob its own learning rate based on how jumpy that knob has been. None of them replace the rule; they decorate it.

The cheat sheet

Three sentences worth keeping

1. The derivative tells you the direction; the learning rate is a guess. One is computed exactly, the other is chosen by a human who will change it later.
2. The steps shrink by themselves. Flat ground means small slope means small step, which is why gradient descent settles instead of rattling around the bottom.
3. Scale changes the bookkeeping, not the move. Whether it is one knob on paper or four hundred billion on a cluster: measure the tilt, take a small step against it, repeat.

That is the whole algorithm. You just ran it.

Related

• SGD vs momentum vs Adam — the refinements, compared.
• The maths you actually need — the four ideas behind all of this.
• Epoch vs batch vs iteration — what one step covers in a real training run.
• Jacobian vs gradient — when the knob becomes billions of knobs.