Task 6 of 6

Going Live, Honestly

Everything you have built runs on a sequence of frames that this module fabricated. Here is what changes when the frames come from a camera instead: nothing, in the kernels. gpu.js accepts an HTMLVideoElement as a kernel argument directly and re-uploads whatever is currently on screen each time you call it. The whole wiring is this:

const video = document.createElement('video');
video.autoplay = true;
video.playsInline = true;
video.srcObject = await navigator.mediaDevices.getUserMedia({ video: true });
await video.play();

const gpu = new GPU();
const filter = gpu.createKernel(function (feed) {
  const p = feed[this.thread.y][this.thread.x];
  const l = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
  this.color(l, l, l, 1);
}, { output: [640, 480], graphical: true });

document.body.appendChild(filter.canvas);

async function tick() {
  await filter(video);          // the current video frame, uploaded for you
  requestAnimationFrame(tick);  // ~16.7 ms later, again
}
requestAnimationFrame(tick);

That code cannot run in this course, and this course is not going to pretend otherwise. Your code here executes inside a Web Worker — that is what lets a runaway kernel be killed instead of freezing the page — and a Worker has no navigator.mediaDevices, no getUserMedia and no HTMLVideoElement. There is no camera to reach and no video element to hand a kernel. The eight-frame sequence is the stand-in, and every line you have written works unchanged the day you paste it onto a page with the loop above.

What is left is the shape of the thing: a filter that is built once and holds its own state, exposing one function you call per frame. Which is where the last wrinkle lives — the first frame. There is no history yet, so the model has to be born from the frame in your hand. Get that wrong and the whole first frame reads as motion, or the state is null and the arithmetic comes back as NaN. Every stateful filter has this three-line initialiser, and it is always the last thing anyone tests.

Array layout in gpu.js

Image data comes in row-major: image[y][x] is the pixel in row y, column x, and each pixel is an [r, g, b, a] array with channels from 0 to 1. Mind the inversion that catches everyone — sizes are given width-first (output: [width, height]), but indexing runs row-first, so this thread's own pixel is image[this.thread.y][this.thread.x]. Swap those two and you read the transpose of your image. Three-dimensional data follows the same rule: output: [w, h, d] is indexed [z][y][x].

Goal: finish onFrame so the filter seeds its model from the first frame only, then detects and learns on every frame after it.

Requirements

Seed model from the first frame only — later frames must not reset it
Return the foreground mask for every frame the filter is given, the first one included
Fold each frame into the model after detecting against it

Hint 1 — "first frame" is a state, not an index

A live filter never sees an array — it is handed one frame and asked for one answer. So "is this the first frame?" has to be a property of the filter, not of a loop counter:

if (model === null) {
  model = await seedModel(image);
}

Which is why model starts as null rather than as a texture.

Hint 2 — what the first frame should report

Seeded from itself, the frame agrees with the model everywhere, so the mask is empty and the count is 0. That is the correct answer, and it is much better than a first frame that lights up completely.

Hint 3 — the three lines

if (model === null) model = await seedModel(image);
const mask = await detect(image, model);
model = await learn(image, model);
return mask;

Detect, then learn. Reversing them lets each frame teach the model about itself before you ask the model what is new.

Same idea elsewhere

Build-once, call-per-frame is the shape of every real-time pipeline: a WebGPU app creates its pipelines and bind groups at startup and only records command buffers inside the frame callback; a CUDA video filter allocates its device buffers and loads its modules once and launches per frame; MediaPipe and OpenCV's VideoCapture loops are the same skeleton. The state that survives between calls — your background model — is the part that makes it a video filter rather than eight unrelated image filters.

All tasks in Video Filters

This page is an interactive exercise — the editor, the GPU runner and your saved progress need JavaScript. The text above is the full brief.