# The Payoff: A Virtual Background

*Task 5 of 6 · [Video Filters](https://gpu.rocks/learn/video-filters-4d39e404.md) · GPU.js Learn*

Time to cash the whole track in. You have a model of the empty scene, and you have
the arithmetic to say how much each pixel disagrees with it. Turn that disagreement into a
**soft mask**, blur a copy of the frame to stand in for a replaced backdrop,
and composite one over the other. That is the effect everybody has seen on a video call, and
it is four kernels.

Two details make it look like a product rather than a demo. The first is that the mask
must be **normalised** — a number from 0 to 1, nothing else. Composite with
`fg × m + bg × (1 − m)` and a mask of 1.4 does not mean "very foreground", it
means the background is subtracted from the picture; a mask of −0.3 means the background is
added twice. So the raw difference gets ramped and clamped:

```js
m = clamp((d - lo) / (hi - lo), 0, 1)
```

The second is the **feather**: the ramp alone gives a hard, jagged edge, and
a 3×3 mean over the mask softens it — the same box blur from Convolution & Filters,
aimed at the mask instead of the picture. It has a second job: an isolated hot pixel that
made it through arrives as a lone 1 and leaves as a 0.11, which is invisible.

Nothing in the chain touches JavaScript. The frame goes up, the model lives on the card,
the mask never comes down, and the graphical pass eats the mask texture and writes pixels.
Readbacks: zero.

**Array layout in gpu.js**
Image data comes in row-major: `image[y][x]` is the pixel in row *y*,
column *x*, and each pixel is an `[r, g, b, a]` array with channels from
0 to 1. Mind the inversion that catches everyone — sizes are given width-first
(`output: [width, height]`), but indexing runs row-first, so this thread's own
pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the
transpose of your image. Three-dimensional data follows the same rule:
`output: [w, h, d]` is indexed `[z][y][x]`.

## Goal

**Goal:** finish the normalised ramp in `softMask` and the
composite line in `compose`, so the foreground stays sharp over a blurred
background.

## Requirements

- In `softMask`, ramp `|now − model|` from `lo` to `hi` and clamp the result into 0…1
- In `compose`, paint `source × m + blurred × (1 − m)` per channel
- Every stage stays on the GPU — the graphical pass is fed the mask *texture*
- Render the result with `render(compose.canvas)`

## Hint 1 — the ramp, normalised

Below `lo` it is all background, above `hi` all
foreground, and in between it slides:

```js
const span = this.constants.hi - this.constants.lo;
return Math.min(Math.max((d - this.constants.lo) / span, 0), 1);
```

The `Math.min`/`Math.max` pair is not decoration — without it the mask
leaves 0…1 and the composite starts subtracting light.

## Hint 2 — the composite

`m = 1` has to give you the source pixel and `m = 0` the
blurred one, so the mask multiplies the *foreground*:

```js
this.color(
  p[0] * m + backR * (1 - m),
  p[1] * m + backG * (1 - m),
  p[2] * m + backB * (1 - m),
  1
);
```

Swap the two and you get a sharp background with a blurry person in it, which is a look, just
not this one.

## Hint 3 — where the background comes from

The 5×5 loop in `compose` is already written: it averages the frame's
own neighbourhood, so the "replaced" backdrop is a blurred copy of the real one. Swap
that average for a fixed colour, or for a second image, and you have a green screen
instead.

## Same idea elsewhere

This is a render graph: named passes, explicit dependencies, every resource resident
on the device — the architecture behind a Frostbite frame graph, a Metal command buffer full
of encoder passes, or CUDA Graphs' pre-recorded launch chains. Shipping virtual backgrounds
replace the luminance model with a segmentation network, but the tail of the pipeline —
ramp, feather, composite — is still exactly these three lines, because it is the part that
has to run in under a millisecond.

## Starter code

```js
const gpu = new GPU({ mode });

// Passes 1 and 2 — the background model, from the last task.
const seedModel = gpu.createKernel(function (frame) {
  const p = frame[this.thread.y][this.thread.x];
  return 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
}, { output: [64, 64], pipeline: true });

const learn = gpu.createKernel(function (frame, model) {
  const p = frame[this.thread.y][this.thread.x];
  const now = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
  return (1 - this.constants.alpha) * model[this.thread.y][this.thread.x]
       + this.constants.alpha * now;
}, { output: [64, 64], pipeline: true, immutable: true, constants: { alpha: 0.05 } });

// Pass 3 — how foreground is this pixel, from 0 to 1?
const softMask = gpu.createKernel(function (frame, model) {
  const p = frame[this.thread.y][this.thread.x];
  const now = 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
  const d = Math.abs(now - model[this.thread.y][this.thread.x]);
  // TODO: ramp d from this.constants.lo to this.constants.hi and CLAMP the
  // result into 0…1. Anything outside that range breaks the composite.
  return d;
}, { output: [64, 64], pipeline: true, constants: { lo: 0.1, hi: 0.22 } });

// Pass 4 — feather the mask. A 3×3 mean softens the cut and demotes any
// surviving speck from 1 to 0.11. (Convolution & Filters' box blur, aimed
// at the mask instead of the picture.)
const feather = gpu.createKernel(function (mask) {
  let sum = 0;
  for (let dy = -1; dy <= 1; dy++) {
    for (let dx = -1; dx <= 1; dx++) {
      let yy = this.thread.y + dy;
      let xx = this.thread.x + dx;
      if (yy < 0) yy = 0;
      if (yy > this.constants.last) yy = this.constants.last;
      if (xx < 0) xx = 0;
      if (xx > this.constants.last) xx = this.constants.last;
      sum += mask[yy][xx];
    }
  }
  return sum / 9;
}, { output: [64, 64], pipeline: true, constants: { last: 63 } });

// Pass 5 — the composite. Texture in, pixels out.
const compose = gpu.createKernel(function (frame, mask) {
  const m = mask[this.thread.y][this.thread.x];
  const p = frame[this.thread.y][this.thread.x];

  // The stand-in backdrop: a 5×5 blur of the frame itself.
  let sr = 0;
  let sg = 0;
  let sb = 0;
  for (let dy = -2; dy <= 2; dy++) {
    for (let dx = -2; dx <= 2; dx++) {
      let yy = this.thread.y + dy;
      let xx = this.thread.x + dx;
      if (yy < 0) yy = 0;
      if (yy > this.constants.last) yy = this.constants.last;
      if (xx < 0) xx = 0;
      if (xx > this.constants.last) xx = this.constants.last;
      const q = frame[yy][xx];
      sr += q[0];
      sg += q[1];
      sb += q[2];
    }
  }
  const backR = sr / 25;
  const backG = sg / 25;
  const backB = sb / 25;

  // TODO: paint the source pixel where m is 1 and the blurred backdrop
  // where m is 0, sliding between them in between.
  this.color(p[0], p[1], p[2], 1);
}, { output: [64, 64], graphical: true, constants: { last: 63 } });

// A dial, not a constant: slider() re-runs the whole program when you drag it,
// so this scrubs the clip — frame 0 is the empty room, frame 7 the one the
// tests check.
const frame = slider('frame', { min: 0, max: frames.length - 1, value: frames.length - 1, step: 1 });

// Build the model from every frame BEFORE this one, then filter this one.
let model = await seedModel(frames[0]);
for (let i = 1; i < frame; i++) {
  model = await learn(frames[i], model);
}

const live = frames[frame];
await compose(live, await feather(await softMask(live, model)));
render(compose.canvas);
```

---

Interactive version: https://gpu.rocks/learn/video-filters-4d39e404/5

[Previous task](https://gpu.rocks/learn/video-filters-4d39e404/4.md) · [Next task](https://gpu.rocks/learn/video-filters-4d39e404/6.md)