# One Equation, Two Unknowns

*Task 1 of 5 · [Optical Flow](https://gpu.rocks/learn/optical-flow-e85c6dfa.md) · GPU.js Learn*

Optical flow asks a simple-sounding question: for every pixel of frame 1, where did
it go in frame 2? The only assumption anyone can make is **brightness constancy** —
a moving point keeps its intensity, it just shows up somewhere else. Write that down and
expand it to first order and you get one equation per pixel:

```js
Ix·u + Iy·v + It = 0
```

`Ix` and `Iy` are the spatial gradients — the same central differences
the Sobel pass in Convolution & Filters is built from — and `It` is how much this
pixel's intensity changed between the frames. `u` and `v` are what you
want, and there is only one equation for the two of them; every method in this module is a
different way of buying a second. One subtlety first, though: all three derivatives have to
describe the *same instant*, the moment halfway between the frames. `It`
naturally sits there, so the spatial gradients are measured on the **average of the two
frames**. Take them from frame 1 alone and the answers scatter — on these frames the
typical error goes from about 0.03 pixels to about 0.19.

The frames arrive as task inputs rather than from a camera, and that is a wall rather than
a shortcut: your code runs inside a Web Worker, which has no `navigator.mediaDevices`,
no `getUserMedia` and no `<video>` element. On an ordinary page the
real thing is short — `getUserMedia` into a `<video>`, then pass
that element straight to a kernel as an argument, which gpu.js accepts as an image source, and
keep the previous frame around. Everything below that point is identical.

**Array layout in gpu.js**
Image data comes in row-major: `image[y][x]` is the pixel in row *y*,
column *x*, and each pixel is an `[r, g, b, a]` array with channels from
0 to 1. Mind the inversion that catches everyone — sizes are given width-first
(`output: [width, height]`), but indexing runs row-first, so this thread's own
pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the
transpose of your image. Three-dimensional data follows the same rule:
`output: [w, h, d]` is indexed `[z][y][x]`.

## Goal

**Goal:** fill a 3-plane derivative field for the pair
`frameA`/`frameB` — plane 0 is `Ix`, plane 1 is
`Iy`, plane 2 is `It`.

## Requirements

- Keep `output: [64, 64, 3]` — the result is indexed `d[z][y][x]`, one plane per derivative
- Plane 0 (`this.thread.z === 0`): `Ix` = `(right − left) / 2`, where *left* and *right* are the **average of the two frames** at `x - 1` and `x + 1`, both clamped to `0…this.constants.last`
- Plane 1: `Iy`, the same central difference down the `y` axis
- Plane 2: `It` = this pixel in the **second** frame minus the same pixel in the first
- The frames are gray, so a pixel's intensity is just its channel 0

## Hint 1 — reading one intensity

An image cell is an `[r, g, b, a]` array, and these frames are gray,
so all you need is the first channel:

```js
const a = frameA[y][x];
const intensity = a[0];
```

## Hint 2 — the horizontal gradient

Clamp the two neighbour columns like every other neighbourhood pass in the
course, average the frames at each of them, then take the central difference:

```js
let left = x - 1;
if (left < 0) left = 0;
let right = x + 1;
if (right > this.constants.last) right = this.constants.last;

const aL = frameA[y][left];
const bL = frameB[y][left];
const aR = frameA[y][right];
const bR = frameB[y][right];
const midLeft = (aL[0] + bL[0]) / 2;
const midRight = (aR[0] + bR[0]) / 2;
return (midRight - midLeft) / 2;
```

## Hint 3 — the temporal one

No neighbours at all — the same pixel, the two frames, second minus first:

```js
const a = frameA[y][x];
const b = frameB[y][x];
return b[0] - a[0];
```

That order is the whole sign convention of this module. Reverse it and every flow
vector you compute from here on points backwards.

## Same idea elsewhere

Packing several per-pixel quantities into the planes of one output is how every
real pipeline does it: a CUDA kernel writes an `float2`/`float4`
surface, a WGSL compute shader writes an `rgba16float` storage texture, Metal
writes an MTLTexture with the gradients in RG and the time difference in B. One pass, one
launch, three fields.

## Starter code

```js
// Three derivative planes for one pair of frames: Ix, Iy, It.
const gpu = new GPU({ mode });

const derivatives = gpu.createKernel(function (frameA, frameB) {
  const x = this.thread.x;
  const y = this.thread.y;

  if (this.thread.z === 0) {
    // TODO: Ix — central difference along x, measured on the AVERAGE
    // of frameA and frameB, neighbour columns clamped to 0…this.constants.last
    return 0;
  }

  if (this.thread.z === 1) {
    // TODO: Iy — the same central difference, but along y
    return 0;
  }

  // TODO: It — this pixel in the SECOND frame minus the same pixel in the first
  return 0;
}, {
  output: [64, 64, 3],
  constants: { last: 63 },
});

const d = await derivatives(frameA, frameB);
console.log('Ix at (45, 30):', d[0][30][45]);
console.log('Iy at (45, 30):', d[1][30][45]);
console.log('It at (45, 30):', d[2][30][45]);
```

---

Interactive version: https://gpu.rocks/learn/optical-flow-e85c6dfa/1

[Next task](https://gpu.rocks/learn/optical-flow-e85c6dfa/2.md)