Task 4 of 5

How Big a Step? Ask 256 at Once

Where did η = 0.1 come from? Nowhere. It was a guess, and guesses are where gradient descent goes wrong: too small and you never arrive, too large and you do not merely overshoot — you diverge, because the step lands you further from the minimum than you started and the next one is longer still.

There is an exact limit. One step turns the error e = θ − θ* into (I − η·H)·e, where H is the matrix of second derivatives, so the walk shrinks the error only while η < 2 / λ_max. On this data H = 2I, which makes λ_max = 2 and the limit exactly η < 1 — and η = 1/2 the rate that lands on the answer in a single step. (A step size with a hard ceiling is not a quirk of optimisation. Explicit time-stepping of a diffusion has one too, for the same eigenvalue reason.)

Believing that is optional; measuring it costs one launch. Give each of 256 threads its own learning rate, let each run a complete 80-step descent on its own copy of m and c, and read all 256 final losses back at once. This is the axis flip that makes hyperparameter search a GPU problem: earlier tasks were parallel over data points, and here the data is small (64 points, the identical loss surface) while the runs are many — so the threads go one per run, and each one walks its own slice of the data by itself.

too short to arrive, exactly right, or thrown out of the bowl
Goal: give each thread its own learning rate and its own descent, and return the final loss — so that rates 1/8 and 1/2 come back at 0.5, rate 1 sits at 25.5 forever, and rate 3/2 is off the scale.

Requirements

Hint 1 — one thread, one rate

This is the same "which element is mine?" question every kernel asks, and the answer is the same: const rate = rates[this.thread.x];. Leave it as rates[0] and all 256 threads run the identical experiment.

Hint 2 — the step
m = m - rate * this.constants.gradScale * gm;
c = c - rate * this.constants.gradScale * gc;

gradScale is 2 / 64, precomputed in JavaScript — two integer constants would divide as integers in the shader and give you zero.

Hint 3 — reading the result

The final losses tell a story in three parts: a band in the middle that reached 0.5, a few rates at the bottom still crawling towards it, and everything past η = 1 heading for infinity. The one at exactly η = 1 is the strangest: it neither converges nor explodes, because |1 − η·λ| = 1 means the error flips sign and keeps its size.

Same idea elsewhere

Sweeping hyperparameters one per thread is the small, cheap version of what a tuner does with whole models across a cluster — and when the per-run state fits in registers there is no reason to leave the GPU between runs at all. The design decision is which axis to parallelise: over data when the data is big, over runs when the runs are many. Same kernel language, opposite layout.

All tasks in Gradient Descent

  1. The Loss You Can Check
  2. The Gradient Is a Reduction
  3. Take the Step
  4. How Big a Step? Ask 256 at Once
  5. A Thousand Starts, Three Answers

This page is an interactive exercise — the editor, the GPU runner and your saved progress need JavaScript. The text above is the full brief.