Task 5 of 6

The Payoff: A Virtual Background

Time to cash the whole track in. You have a model of the empty scene, and you have the arithmetic to say how much each pixel disagrees with it. Turn that disagreement into a soft mask, blur a copy of the frame to stand in for a replaced backdrop, and composite one over the other. That is the effect everybody has seen on a video call, and it is four kernels.

Two details make it look like a product rather than a demo. The first is that the mask must be normalised — a number from 0 to 1, nothing else. Composite with fg × m + bg × (1 − m) and a mask of 1.4 does not mean "very foreground", it means the background is subtracted from the picture; a mask of −0.3 means the background is added twice. So the raw difference gets ramped and clamped:

m = clamp((d - lo) / (hi - lo), 0, 1)

The second is the feather: the ramp alone gives a hard, jagged edge, and a 3×3 mean over the mask softens it — the same box blur from Convolution & Filters, aimed at the mask instead of the picture. It has a second job: an isolated hot pixel that made it through arrives as a lone 1 and leaves as a 0.11, which is invisible.

Nothing in the chain touches JavaScript. The frame goes up, the model lives on the card, the mask never comes down, and the graphical pass eats the mask texture and writes pixels. Readbacks: zero.

Array layout in gpu.js

Image data comes in row-major: image[y][x] is the pixel in row y, column x, and each pixel is an [r, g, b, a] array with channels from 0 to 1. Mind the inversion that catches everyone — sizes are given width-first (output: [width, height]), but indexing runs row-first, so this thread's own pixel is image[this.thread.y][this.thread.x]. Swap those two and you read the transpose of your image. Three-dimensional data follows the same rule: output: [w, h, d] is indexed [z][y][x].

Goal: finish the normalised ramp in softMask and the composite line in compose, so the foreground stays sharp over a blurred background.

Requirements

In softMask, ramp |now − model| from lo to hi and clamp the result into 0…1
In compose, paint source × m + blurred × (1 − m) per channel
Every stage stays on the GPU — the graphical pass is fed the mask texture
Render the result with render(compose.canvas)

Hint 1 — the ramp, normalised

Below lo it is all background, above hi all foreground, and in between it slides:

const span = this.constants.hi - this.constants.lo;
return Math.min(Math.max((d - this.constants.lo) / span, 0), 1);

The Math.min/Math.max pair is not decoration — without it the mask leaves 0…1 and the composite starts subtracting light.

Hint 2 — the composite

m = 1 has to give you the source pixel and m = 0 the blurred one, so the mask multiplies the foreground:

this.color(
  p[0] * m + backR * (1 - m),
  p[1] * m + backG * (1 - m),
  p[2] * m + backB * (1 - m),
  1
);

Swap the two and you get a sharp background with a blurry person in it, which is a look, just not this one.

Hint 3 — where the background comes from

The 5×5 loop in compose is already written: it averages the frame's own neighbourhood, so the "replaced" backdrop is a blurred copy of the real one. Swap that average for a fixed colour, or for a second image, and you have a green screen instead.

Same idea elsewhere

This is a render graph: named passes, explicit dependencies, every resource resident on the device — the architecture behind a Frostbite frame graph, a Metal command buffer full of encoder passes, or CUDA Graphs' pre-recorded launch chains. Shipping virtual backgrounds replace the luminance model with a segmentation network, but the tail of the pipeline — ramp, feather, composite — is still exactly these three lines, because it is the part that has to run in under a millisecond.

All tasks in Video Filters

This page is an interactive exercise — the editor, the GPU runner and your saved progress need JavaScript. The text above is the full brief.