# Going Live, Honestly *Task 6 of 6 · [Video Filters](https://gpu.rocks/learn/video-filters-4d39e404.md) · GPU.js Learn* Everything you have built runs on a sequence of frames that this module fabricated. Here is what changes when the frames come from a camera instead: **nothing, in the kernels**. gpu.js accepts an `HTMLVideoElement` as a kernel argument directly and re-uploads whatever is currently on screen each time you call it. The whole wiring is this: ```js const video = document.createElement('video'); video.autoplay = true; video.playsInline = true; video.srcObject = await navigator.mediaDevices.getUserMedia({ video: true }); await video.play(); const gpu = new GPU(); const filter = gpu.createKernel(function (feed) { const p = feed[this.thread.y][this.thread.x]; const l = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2]; this.color(l, l, l, 1); }, { output: [640, 480], graphical: true }); document.body.appendChild(filter.canvas); async function tick() { await filter(video); // the current video frame, uploaded for you requestAnimationFrame(tick); // ~16.7 ms later, again } requestAnimationFrame(tick); ``` **That code cannot run in this course, and this course is not going to pretend otherwise.** Your code here executes inside a Web Worker — that is what lets a runaway kernel be killed instead of freezing the page — and a Worker has no `navigator.mediaDevices`, no `getUserMedia` and no `HTMLVideoElement`. There is no camera to reach and no video element to hand a kernel. The eight-frame sequence is the stand-in, and every line you have written works unchanged the day you paste it onto a page with the loop above. What is left is the shape of the thing: a filter that is **built once** and holds its own state, exposing one function you call per frame. Which is where the last wrinkle lives — the first frame. There is no history yet, so the model has to be born from the frame in your hand. Get that wrong and the whole first frame reads as motion, or the state is `null` and the arithmetic comes back as `NaN`. Every stateful filter has this three-line initialiser, and it is always the last thing anyone tests. **Array layout in gpu.js** Image data comes in row-major: `image[y][x]` is the pixel in row *y*, column *x*, and each pixel is an `[r, g, b, a]` array with channels from 0 to 1. Mind the inversion that catches everyone — sizes are given width-first (`output: [width, height]`), but indexing runs row-first, so this thread's own pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the transpose of your image. Three-dimensional data follows the same rule: `output: [w, h, d]` is indexed `[z][y][x]`. ## Goal **Goal:** finish `onFrame` so the filter seeds its model from the *first* frame only, then detects and learns on every frame after it. ## Requirements - Seed `model` from the first frame only — later frames must not reset it - Return the foreground mask for every frame the filter is given, the first one included - Fold each frame into the model *after* detecting against it ## Hint 1 — "first frame" is a state, not an index A live filter never sees an array — it is handed one frame and asked for one answer. So "is this the first frame?" has to be a property of the filter, not of a loop counter: ```js if (model === null) { model = await seedModel(image); } ``` Which is why `model` starts as `null` rather than as a texture. ## Hint 2 — what the first frame should report Seeded from itself, the frame agrees with the model everywhere, so the mask is empty and the count is `0`. That is the correct answer, and it is much better than a first frame that lights up completely. ## Hint 3 — the three lines ```js if (model === null) model = await seedModel(image); const mask = await detect(image, model); model = await learn(image, model); return mask; ``` Detect, then learn. Reversing them lets each frame teach the model about itself before you ask the model what is new. ## Same idea elsewhere Build-once, call-per-frame is the shape of every real-time pipeline: a WebGPU app creates its pipelines and bind groups at startup and only records command buffers inside the frame callback; a CUDA video filter allocates its device buffers and loads its modules once and launches per frame; MediaPipe and OpenCV's `VideoCapture` loops are the same skeleton. The state that survives between calls — your background model — is the part that makes it a video filter rather than eight unrelated image filters. ## Starter code ```js // The shippable shape: built once, one call per frame, state kept inside. const gpu = new GPU({ mode }); const seedModel = gpu.createKernel(function (image) { const p = image[this.thread.y][this.thread.x]; return 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2]; }, { output: [64, 64], pipeline: true }); const learn = gpu.createKernel(function (image, model) { const p = image[this.thread.y][this.thread.x]; const now = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2]; return (1 - this.constants.alpha) * model[this.thread.y][this.thread.x] + this.constants.alpha * now; }, { output: [64, 64], pipeline: true, immutable: true, constants: { alpha: 0.05 } }); const detect = gpu.createKernel(function (image, model) { const p = image[this.thread.y][this.thread.x]; const now = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2]; if (Math.abs(now - model[this.thread.y][this.thread.x]) > this.constants.threshold) { return 1; } return 0; }, { output: [64, 64], constants: { threshold: 0.12 } }); // The state that survives between frames. null means "nothing seen yet". let model = null; // One frame in, one mask out. On a page this is what requestAnimationFrame // calls, with the