# Payoff: Thirty-Two Seams *Task 5 of 6 · [Seam Carving: Content-Aware Resizing](https://gpu.rocks/learn/seam-carving-a23a0d9b.md) · GPU.js Learn* One seam is a curiosity. Thirty-two is a resize. And the loop has a catch that matters: once a seam is gone the picture is a *different picture*, so the energy map and the cost map both have to be built again from the carved luminance — not from the original. Energy → cost → seam → carve, and round again. Everything you wrote is here already. What is left is the orchestration, and the order is the whole thing: each stage awaits the one before it, and inside the cost map the rows must go one at a time, because row *y* reads row *y - 1*'s answer. Three colour planes and the luminance plane all reflow with the same seam — the carve kernel has no idea what it is carving, which is why one kernel does all four. Two console tricks pay for themselves here. `render()` once per removal collapses into a **frame scrubber** you can drag back and forth — that is where you actually see the picture reflow. And plotting each seam's energy against its removal number gives you the curve that tells you when to stop: the cheap seams go first, and when the curve knees upwards the picture has run out of things it can afford to lose. The slider under the console re-runs the whole program, so you can carve less and compare. **Array layout in gpu.js** Image data comes in row-major: `image[y][x]` is the pixel in row *y*, column *x*, and each pixel is an `[r, g, b, a]` array with channels from 0 to 1. Mind the inversion that catches everyone — sizes are given width-first (`output: [width, height]`), but indexing runs row-first, so this thread's own pixel is `image[this.thread.y][this.thread.x]`. Swap those two and you read the transpose of your image. Three-dimensional data follows the same rule: `output: [w, h, d]` is indexed `[z][y][x]`. ## Goal **Goal:** fill in `carveOnce` — energy, cumulative cost, backtrack, then carve every plane with that one seam — and return the new planes together with the seam's price. ## Requirements - Build the energy map from the CURRENT luminance plane (`planes[3]`) at the current width — `energy.setOutput([w, 72])` - Build the cumulative rows with one awaited `step` launch per row, in order - `backtrack` the rows to a seam, and take its price from the bottom row - Carve **every** plane with that seam at `[w - 1, 72]`, and return the new planes plus the cost ## Hint 1 — the current width Nothing needs to be tracked by hand: the planes know how wide they are. ```js const w = planes[0][0].length; energy.setOutput([w, 72]); step.setOutput([w]); carve.setOutput([w - 1, 72]); ``` ## Hint 2 — the cost map, again Exactly the driver from the cumulative-cost task, over the energy map you just built: ```js const rows = [Float32Array.from(e[0])]; for (let y = 1; y < 72; y++) rows.push(await step(e[y], rows[y - 1])); ``` ## Hint 3 — carving four planes with one seam A `for` loop, not `.map()` — a callback cannot hold an `await`, and every one of these is a kernel call: ```js const next = []; for (let i = 0; i < planes.length; i++) { next.push(await carve(planes[i], seam)); } return { planes: next, cost: rows[71][seam[71]] }; ``` ## Same idea elsewhere A per-frame chain of dependent passes with a host-side loop around it is what a real pipeline looks like on every platform: CUDA streams a sequence of launches, WebGPU records passes into a command encoder, Metal encodes one compute pass per stage. And the cost of this particular shape is visible in the numbers — 71 cost-map launches per removal, well over two thousand for the whole run, each one tiny. When a wavefront gets slow it is almost never the arithmetic; it is the launch count. ## Starter code ```js // Thirty-two removals, and the picture reflows around what is left. const gpu = new GPU({ mode }); // Drag me: the program is a pure function of its controls, so moving this // re-runs the whole carve. const seams = slider('seams to remove', { min: 20, max: 32, value: 32, step: 1 }); // ImageData in, one numeric plane out: 0/1/2 are r/g/b, 3 is the luminance // the energy map is built from. Four planes that all have to reflow together. const channel = gpu.createKernel(function (image, c) { const p = image[this.thread.y][this.thread.x]; if (c === 0) return p[0]; if (c === 1) return p[1]; if (c === 2) return p[2]; return 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2]; }, { output: [128, 72] }); // Task 1: the Sobel magnitude, border-clamped, width from this.output.x. const energy = gpu.createKernel(function (gray) { const x = this.thread.x; const y = this.thread.y; const xm = Math.max(x - 1, 0); const xp = Math.min(x + 1, this.output.x - 1); const ym = Math.max(y - 1, 0); const yp = Math.min(y + 1, this.output.y - 1); const gx = (gray[ym][xp] + 2 * gray[y][xp] + gray[yp][xp]) - (gray[ym][xm] + 2 * gray[y][xm] + gray[yp][xm]); const gy = (gray[yp][xm] + 2 * gray[yp][x] + gray[yp][xp]) - (gray[ym][xm] + 2 * gray[ym][x] + gray[ym][xp]); return Math.sqrt(gx * gx + gy * gy); }, { output: [128, 72], dynamicOutput: true, dynamicArguments: true }); // Task 2: one row of the cumulative map. One launch per row. const step = gpu.createKernel(function (eRow, prev) { const x = this.thread.x; let best = prev[x]; if (x > 0) best = Math.min(best, prev[x - 1]); if (x + 1 < this.output.x) best = Math.min(best, prev[x + 1]); return eRow[x] + best; }, { output: [128], immutable: true, dynamicOutput: true, dynamicArguments: true }); // Task 4: remove the seam by gathering — one column narrower. const carve = gpu.createKernel(function (plane, seam) { const x = this.thread.x; const y = this.thread.y; if (x < seam[y]) return plane[y][x]; return plane[y][x + 1]; }, { output: [127, 72], immutable: true, dynamicOutput: true, dynamicArguments: true }); // The canvas never shrinks — the PICTURE does. Everything from column w // rightwards is painted as empty frame, so the narrowing is visible. const paint = gpu.createKernel(function (r, g, b, w) { const x = this.thread.x; // this.color() paints from the bottom up — thread.y 0 is the BOTTOM row of // the canvas, on every backend — so read the rows in reverse to put row 0 // back at the top of the picture where it belongs. const y = this.output.y - 1 - this.thread.y; if (x < w) { this.color(r[y][x], g[y][x], b[y][x], 1); } else { this.color(0.09, 0.10, 0.12, 1); } }, { output: [128, 72], graphical: true, dynamicArguments: true }); // Task 3: 72 sequential steps, three numbers each. It stays on the host. function backtrack(cost) { const h = cost.length; const w = cost[0].length; const seam = new Array(h); let x = 0; for (let i = 1; i < w; i++) if (cost[h - 1][i] < cost[h - 1][x]) x = i; seam[h - 1] = x; for (let y = h - 2; y >= 0; y--) { let best = x; if (x > 0 && cost[y][x - 1] < cost[y][best]) best = x - 1; if (x + 1 < w && cost[y][x + 1] < cost[y][best]) best = x + 1; x = best; seam[y] = x; } return seam; } // One removal: energy → cumulative cost → seam → carve every plane. async function carveOnce(planes) { const w = planes[0][0].length; // TODO 1: the energy map of the CURRENT luminance plane, planes[3], at // width w. (energy.setOutput([w, 72]) first.) // TODO 2: the cumulative rows — one awaited step launch per row, in order. // TODO 3: backtrack to a seam, and read its price off the bottom row. // TODO 4: carve every plane in `planes` with that seam, at [w - 1, 72]. return { planes, cost: 0 }; } // r, g, b and the luminance the energy is built from: four planes that all // have to reflow together. let planes = []; for (let c = 0; c < 4; c++) planes.push(await channel(photo, c)); const costs = []; await paint(planes[0], planes[1], planes[2], planes[0][0].length); render(paint.canvas); for (let k = 0; k < seams; k++) { const out = await carveOnce(planes); planes = out.planes; costs.push(out.cost); // one render() per removal — consecutive ones become a frame scrubber await paint(planes[0], planes[1], planes[2], planes[0][0].length); render(paint.canvas); } console.log('carved down to', planes[0][0].length, 'columns'); plot(costs, { title: 'energy of each seam removed', xLabel: 'removal' }); ``` --- Interactive version: https://gpu.rocks/learn/seam-carving-a23a0d9b/5 [Previous task](https://gpu.rocks/learn/seam-carving-a23a0d9b/4.md) · [Next task](https://gpu.rocks/learn/seam-carving-a23a0d9b/6.md)