# The Aperture Problem

*Task 2 of 5 · [Optical Flow](https://gpu.rocks/learn/optical-flow-e85c6dfa.md) · GPU.js Learn*

One equation, two unknowns, is not "nearly enough information". It is a
**line** of answers. `Ix·u + Iy·v + It = 0` is the equation of a
straight line in the `(u, v)` plane, and every point on it fits this pixel's
evidence equally well. Looking harder at the pixel will not narrow it down; the information
is not there.

What *is* there is the component of the motion **along the gradient**.
Perpendicular to the gradient the intensity does not change, so sliding that way is
invisible. Pick the shortest vector on the line and you get the **normal flow**:

```js
u = −It · Ix / (Ix² + Iy²)
v = −It · Iy / (Ix² + Iy²)
```

`derivs` here comes from a pair you can check by hand. The scene is a sawtooth
ramp climbing 12 levels per step along the `(1, 1)` diagonal — so every edge in it
runs anti-diagonally — and frame 2 is that scene moved **4 pixels to the right and 0
down**. You know the answer. The pixel does not, and cannot.

## Figures

- **the window cannot tell these apart — only the across-the-edge part is real** — An anti-diagonal edge seen through a small circular aperture. The true motion arrow points right; the only recoverable component points down-right, at right angles to the edge. A dashed line through both arrowheads marks the whole family of motions that fit the same evidence.

## Goal

**Goal:** compute the normal flow for every pixel — plane 0 is
`u`, plane 1 is `v` — and watch it fail to find a motion you know
exactly.

## Requirements

- Keep `output: [64, 64, 2]`: plane 0 is `u`, plane 1 is `v`
- Read this pixel's three derivatives from `derivs[0][y][x]`, `derivs[1][y][x]`, `derivs[2][y][x]`
- Return `-It * Ix / (Ix*Ix + Iy*Iy)` for plane 0 and `-It * Iy / (Ix*Ix + Iy*Iy)` for plane 1
- **Sign convention:** `(u, v)` is where the content *went*, in pixels per frame — positive `u` is rightward, positive `v` is downward. That is the sign that makes `Ix·u + Iy·v + It` come out zero

## Hint 1 — one pixel, no neighbours

Nothing is gathered here: thread `(x, y)` reads exactly three numbers
and returns one. The whole body fits in six lines.

## Hint 2 — name the squared gradient once

```js
const ix = derivs[0][y][x];
const iy = derivs[1][y][x];
const it = derivs[2][y][x];
const g = ix * ix + iy * iy;
```

then plane 0 returns `(-it * ix) / g` and plane 1 returns
`(-it * iy) / g`. This pair has a gradient at every pixel, so no guard is
needed yet — task 3 is where that stops being true.

## Same idea elsewhere

Normal flow is not a toy: it is what a single-pixel constraint can honestly give
you, and it is the per-thread starting point every dense-flow implementation refines —
CUDA's `NVOF` optical-flow engine, OpenCV's `calcOpticalFlowFarneback`,
the motion-vector passes inside DLSS and FSR. All of them add an assumption on top; none of
them can conjure the missing component out of one pixel.

## Starter code

```js
// Normal flow: the shortest vector that satisfies one pixel's equation.
const gpu = new GPU({ mode });

const normalFlow = gpu.createKernel(function (derivs) {
  const x = this.thread.x;
  const y = this.thread.y;
  const ix = derivs[0][y][x];
  const iy = derivs[1][y][x];
  const it = derivs[2][y][x];

  // TODO: plane 0 (this.thread.z === 0) returns u, plane 1 returns v.
  // u = -it * ix / (ix*ix + iy*iy),  v = -it * iy / (ix*ix + iy*iy)
  return 0;
}, {
  output: [64, 64, 2],
});

const flow = await normalFlow(derivs);
console.log('true motion: (4, 0)');
console.log('estimated at (30, 21):', flow[0][21][30], flow[1][21][30]);
```

---

Interactive version: https://gpu.rocks/learn/optical-flow-e85c6dfa/2

[Previous task](https://gpu.rocks/learn/optical-flow-e85c6dfa/1.md) · [Next task](https://gpu.rocks/learn/optical-flow-e85c6dfa/3.md)