# Sixteen Milliseconds

*Task 1 of 6 · [Video Filters](https://gpu.rocks/learn/video-filters-4d39e404.md) · GPU.js Learn*

Everything so far in this track processed *one* picture. Video changes the
economics completely. At 60 frames per second you get **16.7 milliseconds** to
do all of it — decode, filter, composite, present — and then the frame is gone whether you
were finished or not. Miss the budget and you do not get a slower filter, you get a stuttering
one.

The same three kernels now run eight times instead of once, which turns two habits that
were merely wasteful into the whole problem. The first is creating kernels inside the loop:
`createKernel` transpiles your JavaScript to shader source and compiles it, so
doing it per frame pays the compiler sixty times a second. The second is the readback —
Pipelines & Textures showed the mechanism, and here it is the difference between a filter
that runs and one that does not.

The starter below is honest, working, and unshippable. Fix its structure, then read the
numbers it prints — and hit **⏱ Benchmark** afterwards to see the same argument
made twice.

**Array layout in gpu.js**
Image data comes in row-major: `image[y][x]` is the pixel in row *y*,
column *x*, and each pixel is an `[r, g, b, a]` array with channels from
0 to 1. Mind the inversion that catches everyone — sizes are given width-first
(`output: [width, height]`), but indexing runs row-first, so this thread's own
pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the
transpose of your image. Three-dimensional data follows the same rule:
`output: [w, h, d]` is indexed `[z][y][x]`.

## Figures

- **same three kernels, same arithmetic — the readbacks are what miss the frame**

## Goal

**Goal:** hoist the three kernels out of the frame loop, keep the two
intermediate stages on the GPU, and report the per-frame cost against the 16.7 ms budget.

## Requirements

- Create the three kernels **once**, above the loop — exactly three for the whole run
- Give `luminance` and `denoise` `pipeline: true`; `tone` stays plain
- Process every frame in `frames`, with no `.toArray()` inside the loop
- Log `'processed', out.length, 'frames'` and a per-frame line carrying the ms and the fps

## Hint 1 — what belongs in the loop

Everything that depends on *which* frame you are looking at, and nothing
else. A kernel does not depend on the frame — it takes one as an argument. Three
`createKernel` calls, then a loop that does nothing but call them.

## Hint 2 — where the readbacks are hiding

There is no `.toArray()` in the starter, and the readbacks are still
there: a non-pipeline kernel's *return value* is the readback. Add
`pipeline: true` to the first two stages and the chain collapses to one
expression:

```js
out.push(await tone(await denoise(await luminance(frames[i]))));
```

## Hint 3 — the budget arithmetic

One frame at 60 fps is `1000 / 60 = 16.7` ms. So:

```js
const perFrame = totalMs / frames.length;
const fps = 1000 / perFrame;
```

and the verdict is just `perFrame <= 16.7`.

## Same idea elsewhere

Every real-time GPU API separates "build the pipeline" from "run it", precisely so
the expensive half happens once: WebGPU's `createRenderPipeline` versus
`dispatchWorkgroups`, Vulkan's pipeline objects, CUDA modules loaded once and
launched forever. And the per-frame budget is why frame graphs exist at all — a readback
mid-frame is a full pipeline stall on every one of them.

## Starter code

```js
// Eight frames, three stages each. Works. Would never ship.
const gpu = new GPU({ mode });

const t0 = performance.now();
const out = [];

for (let i = 0; i < frames.length; i++) {
  // TODO: createKernel transpiles your function and compiles a shader.
  // Doing it here pays that bill on every single frame — hoist all three
  // of these above the loop so they are built once and called eight times.
  const luminance = gpu.createKernel(function (frame) {
    const p = frame[this.thread.y][this.thread.x];
    return 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
  }, { output: [64, 64] });

  const denoise = gpu.createKernel(function (map) {
    let sum = 0;
    for (let dy = -1; dy <= 1; dy++) {
      for (let dx = -1; dx <= 1; dx++) {
        let yy = this.thread.y + dy;
        let xx = this.thread.x + dx;
        if (yy < 0) yy = 0;
        if (yy > this.constants.last) yy = this.constants.last;
        if (xx < 0) xx = 0;
        if (xx > this.constants.last) xx = this.constants.last;
        sum += map[yy][xx];
      }
    }
    return sum / 9;
  }, { output: [64, 64], constants: { last: 63 } });

  const tone = gpu.createKernel(function (map) {
    const v = map[this.thread.y][this.thread.x];
    return Math.min(Math.max((v - 0.35) * 1.8 + 0.5, 0), 1);
  }, { output: [64, 64] });

  // TODO: each of these three stages ends in a download and the next one
  // re-uploads. Only the LAST stage should come back to JavaScript — make
  // the first two pipeline kernels.
  const lum = await luminance(frames[i]);
  const clean = await denoise(lum);
  out.push(await tone(clean));
}

const totalMs = performance.now() - t0;

// TODO: 60 fps is one frame every 16.7 ms. Work out the per-frame cost,
// log it with the frame rate it implies, and say whether it fits:
//   console.log('processed', out.length, 'frames');
//   console.log('per frame:', perFrame.toFixed(2), 'ms -', fps.toFixed(0), 'fps');
console.log('total:', totalMs.toFixed(2), 'ms');
```

---

Interactive version: https://gpu.rocks/learn/video-filters-4d39e404/1

[Next task](https://gpu.rocks/learn/video-filters-4d39e404/2.md)