Gradient Descent Is Just Walking Downhill Blindfolded

· 3 min read · Syed Omar Faruk Towaha
Gradient Descent Is Just Walking Downhill Blindfolded

Imagine you're on a hill in thick fog. You want to reach the lowest point of the valley, but you can't see more than a step ahead. What do you do?

You feel the ground with your feet, figure out which direction slopes down most steeply, and take a step that way. Then you do it again. And again. Eventually, you reach the bottom (or at least a bottom).

That's gradient descent. That's genuinely most of it.

Translating the hill into maths

Each step, you update the weights a little in the direction that reduces the loss:

weights = weights - learning_rate * gradient

A tiny example you can run

Fit a line y = w * x to some points:

import numpy as np

x = np.array([1, 2, 3, 4, 5], dtype=float)
y = np.array([2.1, 3.9, 6.2, 8.1, 9.8])   # roughly y = 2x

w = 0.0
lr = 0.01
for step in range(200):
    pred = w * x
    grad = 2 * np.mean((pred - y) * x)   # derivative of mean squared error
    w -= lr * grad

print(round(w, 3))   # close to 2

No magic. Just "which way is downhill?", over and over.

The learning rate is everything

Learning rates
Too shy and you never arrive. Too brave and you bounce out of the valley.

Real-world upgrades

Plain gradient descent has been upgraded many times:

The honest caveat

Real loss landscapes have millions of dimensions, flat plateaus and many valleys. Gradient descent doesn't guarantee the best valley, only a reasonably low one. Surprisingly, for large neural networks, "reasonably low" turns out to be very good.

So next time someone says a model "learned," picture a blindfolded hiker on a foggy hill, carefully feeling for the downward slope, billions of times. It's less mystical than it sounds, and much more impressive.

// related

// prefer the terminal?

Open the terminal blog and type read gradient-descent-walking-downhill-blindfolded.