# What Moved?

*Task 3 of 6 · [Video Filters](https://gpu.rocks/learn/video-filters-4d39e404.md) · GPU.js Learn*

Subtract one frame from the one before it. Everything that stayed put cancels to
roughly zero; everything that moved does not. Threshold what is left and you have a
**motion mask** — one bit per pixel, "something happened here" — and that is
the first step of essentially every "is anything moving?" system ever shipped, from a
doorbell camera to a video codec deciding which blocks to re-encode.

Two things make or break it. The first is the absolute value: a pixel that got
*darker* moved exactly as much as one that got brighter, and dropping
`Math.abs` silently throws away half of every edge — the mask still looks
plausible, which is what makes it nasty. The second is noise. A raw thresholded difference
is speckled with isolated pixels that the sensor invented, so the mask gets a cleanup pass:
a 3×3 **majority vote**, in the spirit of the morphological open you met in
Thresholding & Morphology. A lone hot pixel has one vote out of nine and loses. The
inside of something that really moved has nine and does not.

And frame 0 has no predecessor. Eight frames give you **seven** differences,
not eight — the classic off-by-one at the start of a sequence, and the reason so many
filters flash garbage on their very first frame.

**Array layout in gpu.js**
Image data comes in row-major: `image[y][x]` is the pixel in row *y*,
column *x*, and each pixel is an `[r, g, b, a]` array with channels from
0 to 1. Mind the inversion that catches everyone — sizes are given width-first
(`output: [width, height]`), but indexing runs row-first, so this thread's own
pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the
transpose of your image. Three-dimensional data follows the same rule:
`output: [w, h, d]` is indexed `[z][y][x]`.

## Goal

**Goal:** finish the `motion` kernel — the absolute luminance
difference between two frames, thresholded to 1 or 0 — and hand the right pair of frames to
it.

## Requirements

- In `motion`, take the luminance of both frames and the **absolute** difference
- Return `1` when that difference exceeds `this.constants.threshold`, otherwise `0`
- Compare each frame with the one *before* it — seven differences from eight frames

## Hint 1 — the difference, both ways

Motion is a change in either direction:

```js
const change = Math.abs(now - before);
if (change > this.constants.threshold) {
  return 1;
}
return 0;
```

Drop the `Math.abs` and the trailing edge of every moving object disappears.

## Hint 2 — which pair of frames

`previous` has to be the frame *before* this one:
`frames[i - 1]`. Hand the kernel `frames[i]` twice and the
difference is zero everywhere — a perfectly quiet, perfectly useless mask.

## Hint 3 — where the loop starts

`frames[i - 1]` only exists from `i = 1` onwards, so the
loop starts there. Eight frames, seven differences — the log line prints the count so you
can see it.

## Same idea elsewhere

Frame differencing is the oldest trick in video and still the load-bearing one.
Motion estimation in H.264/AV1 starts from exactly this residual; OpenCV ships
`absdiff` plus a threshold as the canonical first example; and every "smart"
security camera on the market is this kernel plus a blob counter. On any GPU it is a
one-instruction-per-pixel pass whose real cost is getting the two frames resident at once.

## Starter code

```js
// Two frames in, one bit per pixel out.
const gpu = new GPU({ mode });

const motion = gpu.createKernel(function (current, previous) {
  const a = current[this.thread.y][this.thread.x];
  const b = previous[this.thread.y][this.thread.x];
  const now = 0.299 * a[0] + 0.587 * a[1] + 0.114 * a[2];
  const before = 0.299 * b[0] + 0.587 * b[1] + 0.114 * b[2];
  // TODO: how far did this pixel move, in EITHER direction? Return 1 when
  // that exceeds this.constants.threshold, and 0 when it does not.
  return 0;
}, {
  output: [64, 64],
  pipeline: true,
  constants: { threshold: 0.12 },
});

// The cleanup pass: a 3×3 majority vote. Five of nine neighbours have to
// agree before a pixel stays lit, so isolated sensor noise loses and the
// inside of a real moving blob does not. (Given — you wrote this shape in
// Thresholding & Morphology.)
const cleanup = gpu.createKernel(function (mask) {
  let votes = 0;
  for (let dy = -1; dy <= 1; dy++) {
    for (let dx = -1; dx <= 1; dx++) {
      let yy = this.thread.y + dy;
      let xx = this.thread.x + dx;
      if (yy < 0) yy = 0;
      if (yy > this.constants.last) yy = this.constants.last;
      if (xx < 0) xx = 0;
      if (xx > this.constants.last) xx = this.constants.last;
      votes += mask[yy][xx];
    }
  }
  if (votes >= 5) {
    return 1;
  }
  return 0;
}, { output: [64, 64], constants: { last: 63 } });

const masks = [];
// TODO: frame 0 has no predecessor. Where does this loop really start,
// and which frame belongs in `previous`?
for (let i = 0; i < frames.length; i++) {
  const previous = frames[i];
  masks.push(await cleanup(await motion(frames[i], previous)));
}

console.log('motion masks:', masks.length);

const last = masks[masks.length - 1];
let moving = 0;
for (let y = 0; y < 64; y++) {
  for (let x = 0; x < 64; x++) moving += last[y][x];
}
console.log('moving pixels in the last mask:', moving);
```

---

Interactive version: https://gpu.rocks/learn/video-filters-4d39e404/3

[Previous task](https://gpu.rocks/learn/video-filters-4d39e404/2.md) · [Next task](https://gpu.rocks/learn/video-filters-4d39e404/4.md)