# Learning the Empty Room

*Task 4 of 6 · [Video Filters](https://gpu.rocks/learn/video-filters-4d39e404.md) · GPU.js Learn*

Frame differencing has a blind spot you can see in its own output: it only ever
finds the *edges* of a moving object. The middle of a large uniform blob looks
identical from one frame to the next, so it reports as still. And an object that stops
moving vanishes entirely.

The fix is to stop comparing against the last frame and start comparing against a
**model of the empty scene** — an estimate of what each pixel looks like when
nothing is happening there. Keep that model as a running average with a *very* small
`alpha`, the same one-liner as the last task with the dial turned right down:

```js
model = (1 - alpha) * model + alpha * now
```

At `alpha = 0.05` the model needs about twenty frames to accept a change, so
an object crossing the frame in eight never gets absorbed — but the sun going behind a
cloud eventually does. That is the entire trade, and it is worth saying out loud:
**too fast** and a person who stops moving is quietly re-labelled as furniture;
**too slow** and every genuine change — a chair moved, a light switched on —
leaves a ghost burning in the mask for a minute. Nobody has a principled way to pick it.
People measure.

Then foreground is whatever the current frame disagrees with the model about — the same
absolute difference and threshold as before, against a different reference. Watch what comes
out: a solid object, not a pair of crescents.

**Array layout in gpu.js**
Image data comes in row-major: `image[y][x]` is the pixel in row *y*,
column *x*, and each pixel is an `[r, g, b, a]` array with channels from
0 to 1. Mind the inversion that catches everyone — sizes are given width-first
(`output: [width, height]`), but indexing runs row-first, so this thread's own
pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the
transpose of your image. Three-dimensional data follows the same rule:
`output: [w, h, d]` is indexed `[z][y][x]`.

## Figures

- **subtract the room you already know, and what is left is what arrived**

## Goal

**Goal:** write the exponential update in `learn` and the
subtraction in `foreground`, then run the model over the sequence.

## Requirements

- In `learn`, return `(1 - alpha) * model + alpha * now` — `alpha` on the new frame
- In `foreground`, return `1` when `|now - model|` exceeds the threshold, else `0`
- Segment each frame against the model as it stands, *then* fold that frame in

## Hint 1 — which term wears the alpha

The model is mostly memory and only slightly news, so the big weight sits on the
model:

```js
return (1 - this.constants.alpha) * model[this.thread.y][this.thread.x]
     + this.constants.alpha * now;
```

Put `alpha` on the wrong term and the model becomes the current frame in about one
frame flat — after which nothing is ever foreground again.

## Hint 2 — the subtraction

Identical in shape to the frame difference from the last task, only the
reference changed:

```js
if (Math.abs(now - model[this.thread.y][this.thread.x]) > this.constants.threshold) {
  return 1;
}
return 0;
```

## Hint 3 — order inside the loop

Detect first, learn second. If you fold the frame in before you compare against
it, the model has already moved a little way toward the object you are trying to find —
you are grading your own homework.

## Same idea elsewhere

Every serious background subtractor is this line with more machinery on top: OpenCV's
`MOG2` keeps a mixture of Gaussians per pixel instead of one mean,
`KNN` keeps a sample history, and both still expose a learning rate that behaves
exactly like this `alpha`. On a GPU the appeal never changes — one number of state
per pixel, one multiply-add per frame, perfectly parallel, and no history buffer to carry.

## Starter code

```js
// A model of the empty room, updated a little at a time.
const gpu = new GPU({ mode });

// Seed the model with the first frame you are given. (Frame 0 of this
// sequence is a clean plate — the object is still off the left edge.)
const seedModel = gpu.createKernel(function (frame) {
  const p = frame[this.thread.y][this.thread.x];
  return 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
}, { output: [64, 64], pipeline: true });

// Fold one frame into the model.
const learn = gpu.createKernel(function (frame, model) {
  const p = frame[this.thread.y][this.thread.x];
  const now = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
  // TODO: keep (1 - alpha) of the model and take alpha of `now`.
  return model[this.thread.y][this.thread.x];
}, {
  output: [64, 64],
  pipeline: true,
  immutable: true,
  constants: { alpha: 0.05 },
});

// Anything this frame disagrees with the model about is foreground.
const foreground = gpu.createKernel(function (frame, model) {
  const p = frame[this.thread.y][this.thread.x];
  const now = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
  // TODO: 1 where |now - model| exceeds this.constants.threshold, else 0.
  return 0;
}, {
  output: [64, 64],
  constants: { threshold: 0.12 },
});

let model = await seedModel(frames[0]);
let mask = null;
for (let i = 1; i < frames.length; i++) {
  mask = await foreground(frames[i], model); // detect against the model as it stands
  model = await learn(frames[i], model);     // then let it learn this frame
}

let count = 0;
for (let y = 0; y < 64; y++) {
  for (let x = 0; x < 64; x++) count += mask[y][x];
}
console.log('foreground pixels in the last frame:', count);
```

---

Interactive version: https://gpu.rocks/learn/video-filters-4d39e404/4

[Previous task](https://gpu.rocks/learn/video-filters-4d39e404/3.md) · [Next task](https://gpu.rocks/learn/video-filters-4d39e404/5.md)