Task 4 of 6

Learning the Empty Room

Frame differencing has a blind spot you can see in its own output: it only ever finds the edges of a moving object. The middle of a large uniform blob looks identical from one frame to the next, so it reports as still. And an object that stops moving vanishes entirely.

The fix is to stop comparing against the last frame and start comparing against a model of the empty scene — an estimate of what each pixel looks like when nothing is happening there. Keep that model as a running average with a very small alpha, the same one-liner as the last task with the dial turned right down:

model = (1 - alpha) * model + alpha * now

At alpha = 0.05 the model needs about twenty frames to accept a change, so an object crossing the frame in eight never gets absorbed — but the sun going behind a cloud eventually does. That is the entire trade, and it is worth saying out loud: too fast and a person who stops moving is quietly re-labelled as furniture; too slow and every genuine change — a chair moved, a light switched on — leaves a ghost burning in the mask for a minute. Nobody has a principled way to pick it. People measure.

Then foreground is whatever the current frame disagrees with the model about — the same absolute difference and threshold as before, against a different reference. Watch what comes out: a solid object, not a pair of crescents.

Array layout in gpu.js

Image data comes in row-major: image[y][x] is the pixel in row y, column x, and each pixel is an [r, g, b, a] array with channels from 0 to 1. Mind the inversion that catches everyone — sizes are given width-first (output: [width, height]), but indexing runs row-first, so this thread's own pixel is image[this.thread.y][this.thread.x]. Swap those two and you read the transpose of your image. Three-dimensional data follows the same rule: output: [w, h, d] is indexed [z][y][x].

subtract the room you already know, and what is left is what arrived

Goal: write the exponential update in learn and the subtraction in foreground, then run the model over the sequence.

Requirements

In learn, return (1 - alpha) * model + alpha * now — alpha on the new frame
In foreground, return 1 when |now - model| exceeds the threshold, else 0
Segment each frame against the model as it stands, then fold that frame in

Hint 1 — which term wears the alpha

The model is mostly memory and only slightly news, so the big weight sits on the model:

return (1 - this.constants.alpha) * model[this.thread.y][this.thread.x]
     + this.constants.alpha * now;

Put alpha on the wrong term and the model becomes the current frame in about one frame flat — after which nothing is ever foreground again.

Hint 2 — the subtraction

Identical in shape to the frame difference from the last task, only the reference changed:

if (Math.abs(now - model[this.thread.y][this.thread.x]) > this.constants.threshold) {
  return 1;
}
return 0;

Hint 3 — order inside the loop

Detect first, learn second. If you fold the frame in before you compare against it, the model has already moved a little way toward the object you are trying to find — you are grading your own homework.

Same idea elsewhere

Every serious background subtractor is this line with more machinery on top: OpenCV's MOG2 keeps a mixture of Gaussians per pixel instead of one mean, KNN keeps a sample history, and both still expose a learning rate that behaves exactly like this alpha. On a GPU the appeal never changes — one number of state per pixel, one multiply-add per frame, perfectly parallel, and no history buffer to carry.

All tasks in Video Filters

This page is an interactive exercise — the editor, the GPU runner and your saved progress need JavaScript. The text above is the full brief.