Task 1 of 5

One Equation, Two Unknowns

Optical flow asks a simple-sounding question: for every pixel of frame 1, where did it go in frame 2? The only assumption anyone can make is brightness constancy — a moving point keeps its intensity, it just shows up somewhere else. Write that down and expand it to first order and you get one equation per pixel:

Ix·u + Iy·v + It = 0

Ix and Iy are the spatial gradients — the same central differences the Sobel pass in Convolution & Filters is built from — and It is how much this pixel's intensity changed between the frames. u and v are what you want, and there is only one equation for the two of them; every method in this module is a different way of buying a second. One subtlety first, though: all three derivatives have to describe the same instant, the moment halfway between the frames. It naturally sits there, so the spatial gradients are measured on the average of the two frames. Take them from frame 1 alone and the answers scatter — on these frames the typical error goes from about 0.03 pixels to about 0.19.

The frames arrive as task inputs rather than from a camera, and that is a wall rather than a shortcut: your code runs inside a Web Worker, which has no navigator.mediaDevices, no getUserMedia and no <video> element. On an ordinary page the real thing is short — getUserMedia into a <video>, then pass that element straight to a kernel as an argument, which gpu.js accepts as an image source, and keep the previous frame around. Everything below that point is identical.

Array layout in gpu.js

Image data comes in row-major: image[y][x] is the pixel in row y, column x, and each pixel is an [r, g, b, a] array with channels from 0 to 1. Mind the inversion that catches everyone — sizes are given width-first (output: [width, height]), but indexing runs row-first, so this thread's own pixel is image[this.thread.y][this.thread.x]. Swap those two and you read the transpose of your image. Three-dimensional data follows the same rule: output: [w, h, d] is indexed [z][y][x].

Goal: fill a 3-plane derivative field for the pair frameA/frameB — plane 0 is Ix, plane 1 is Iy, plane 2 is It.

Requirements

Keep output: [64, 64, 3] — the result is indexed d[z][y][x], one plane per derivative
Plane 0 (this.thread.z === 0): Ix = (right − left) / 2, where left and right are the average of the two frames at x - 1 and x + 1, both clamped to 0…this.constants.last
Plane 1: Iy, the same central difference down the y axis
Plane 2: It = this pixel in the second frame minus the same pixel in the first
The frames are gray, so a pixel's intensity is just its channel 0

Hint 1 — reading one intensity

An image cell is an [r, g, b, a] array, and these frames are gray, so all you need is the first channel:

const a = frameA[y][x];
const intensity = a[0];

Hint 2 — the horizontal gradient

Clamp the two neighbour columns like every other neighbourhood pass in the course, average the frames at each of them, then take the central difference:

let left = x - 1;
if (left < 0) left = 0;
let right = x + 1;
if (right > this.constants.last) right = this.constants.last;

const aL = frameA[y][left];
const bL = frameB[y][left];
const aR = frameA[y][right];
const bR = frameB[y][right];
const midLeft = (aL[0] + bL[0]) / 2;
const midRight = (aR[0] + bR[0]) / 2;
return (midRight - midLeft) / 2;

Hint 3 — the temporal one

No neighbours at all — the same pixel, the two frames, second minus first:

const a = frameA[y][x];
const b = frameB[y][x];
return b[0] - a[0];

That order is the whole sign convention of this module. Reverse it and every flow vector you compute from here on points backwards.

Same idea elsewhere

Packing several per-pixel quantities into the planes of one output is how every real pipeline does it: a CUDA kernel writes an float2/float4 surface, a WGSL compute shader writes an rgba16float storage texture, Metal writes an MTLTexture with the gradients in RG and the time difference in B. One pass, one launch, three fields.

All tasks in Optical Flow

This page is an interactive exercise — the editor, the GPU runner and your saved progress need JavaScript. The text above is the full brief.