# What It Does Badly, and the Fix Everybody Ships

*Task 6 of 6 · [Seam Carving: Content-Aware Resizing](https://gpu.rocks/learn/seam-carving-a23a0d9b.md) · GPU.js Learn*

Drag the last task's scrubber slowly and watch the face. The eyes creep together,
the head goes oval, and by the end it is a different person: every row of it wide enough to
matter loses exactly seven columns, so its widest row comes back thirty columns instead of
thirty-seven — and not the same seven in every row (twenty-eight different columns of the
face are cut somewhere), so its features stop lining up vertically. That skew is the straight-line artefact seam carving is famous for,
showing up here as a warp. Seam carving has no idea what a face is. It knows that skin is
smooth and that smooth is cheap, so once the sky is used up the face is the next best
bargain in the picture.

The two poles, meanwhile, come through perfectly straight — every seam in this run
happened to pass entirely to one side of them. That is luck, not a property, and it is the
uncomfortable part: which structures survive depends on where the cheap material happens to
be, and you cannot read it off the picture beforehand.

This is why nothing ships it unattended. Every product that offers content-aware resize
offers a brush next to it, and the brush paints a **mask**: a region whose
energy gets a large constant added, so every seam routes around it. Twenty is plenty here —
the dearest seam this picture has anywhere in the run prices at under nineteen, and the ones
actually taken top out at fourteen, so a single protected pixel already prices a seam out of
the neighbourhood.

One trap comes free with the idea, and it is the reason the mask is carried in
`planes` with everything else: the mask lives in *image* space. Carve the
picture without carving the mask and the protection slides off the thing it was protecting,
one column at a time.

## Goal

**Goal:** write `maskedEnergy`, then measure the result — count
the protected pixels that survived, and plot the protected run's costs against the
unprotected ones.

## Requirements

- `maskedEnergy` returns the Sobel magnitude *plus* `this.constants.penalty * mask[y][x]`
- Count the 1s left in the carved mask (`planes[4]`) after the run and `console.log` it — it should equal what you started with
- Plot both curves together: `plot({ ... })` with `unmaskedCosts` and your own `costs`

## Hint 1 — the penalty term

The Sobel part is untouched; the mask is one more term on the end:

```js
return Math.sqrt(gx * gx + gy * gy)
     + this.constants.penalty * mask[y][x];
```

Because the mask is 0 outside the protected region, this costs the rest of the picture
exactly nothing.

## Hint 2 — counting what survived

`planes[4]` is the mask after the same 32 carves as the picture, so
it is a plain 2D array one column narrower per removal:

```js
let left = 0;
for (let y = 0; y < 72; y++) {
  for (let x = 0; x < planes[4][y].length; x++) left += planes[4][y][x];
}
```

If that number has dropped, a seam went through the face.

## Hint 3 — two series in one chart

`plot` takes an object of named series and draws them on the same
axes:

```js
plot({ 'no mask': unmaskedCosts, 'face protected': costs },
     { title: 'what protection costs', xLabel: 'removal' });
```

## Same idea elsewhere

Weighted energy is how this is done everywhere — Photoshop's content-aware scale
takes a protect/remove mask, and the same "add a large constant to the cost field" move
drives graph-cut segmentation, path planning around obstacles, and every route planner that
has ever been told to avoid motorways. The deeper lesson is the honest one: an algorithm
that optimises a proxy will happily destroy whatever the proxy does not measure, and the
fix is always to put the missing knowledge into the objective rather than to hope.

## Starter code

```js
// The same carve, with a brush stroke over the face.
const gpu = new GPU({ mode });

// ImageData in, one numeric plane out: 0/1/2 are r/g/b, 3 is the luminance
// the energy map is built from. Four planes that all have to reflow together.
const channel = gpu.createKernel(function (image, c) {
  const p = image[this.thread.y][this.thread.x];
  if (c === 0) return p[0];
  if (c === 1) return p[1];
  if (c === 2) return p[2];
  return 0.299 * p[0] + 0.587 * p[1] + 0.114 * p[2];
}, { output: [128, 72] });

// The energy map, plus one term. Everything else is task 1's kernel.
const maskedEnergy = gpu.createKernel(function (gray, mask) {
  const x = this.thread.x;
  const y = this.thread.y;
  const xm = Math.max(x - 1, 0);
  const xp = Math.min(x + 1, this.output.x - 1);
  const ym = Math.max(y - 1, 0);
  const yp = Math.min(y + 1, this.output.y - 1);
  const gx = (gray[ym][xp] + 2 * gray[y][xp] + gray[yp][xp])
           - (gray[ym][xm] + 2 * gray[y][xm] + gray[yp][xm]);
  const gy = (gray[yp][xm] + 2 * gray[yp][x] + gray[yp][xp])
           - (gray[ym][xm] + 2 * gray[ym][x] + gray[ym][xp]);
  // TODO 1: add this.constants.penalty * mask[y][x] to the magnitude, so a
  //         seam that clips the protected region prices itself out.
  return Math.sqrt(gx * gx + gy * gy);
}, {
  output: [128, 72],
  constants: { penalty: 20 },
  dynamicOutput: true,
  dynamicArguments: true,
});

// Task 2: one row of the cumulative map. One launch per row.
const step = gpu.createKernel(function (eRow, prev) {
  const x = this.thread.x;
  let best = prev[x];
  if (x > 0) best = Math.min(best, prev[x - 1]);
  if (x + 1 < this.output.x) best = Math.min(best, prev[x + 1]);
  return eRow[x] + best;
}, { output: [128], immutable: true, dynamicOutput: true, dynamicArguments: true });

// Task 4: remove the seam by gathering — one column narrower.
const carve = gpu.createKernel(function (plane, seam) {
  const x = this.thread.x;
  const y = this.thread.y;
  if (x < seam[y]) return plane[y][x];
  return plane[y][x + 1];
}, { output: [127, 72], immutable: true, dynamicOutput: true, dynamicArguments: true });

// The canvas never shrinks — the PICTURE does. Everything from column w
// rightwards is painted as empty frame, so the narrowing is visible.
const paint = gpu.createKernel(function (r, g, b, w) {
  const x = this.thread.x;
  // this.color() paints from the bottom up — thread.y 0 is the BOTTOM row of
  // the canvas, on every backend — so read the rows in reverse to put row 0
  // back at the top of the picture where it belongs.
  const y = this.output.y - 1 - this.thread.y;
  if (x < w) {
    this.color(r[y][x], g[y][x], b[y][x], 1);
  } else {
    this.color(0.09, 0.10, 0.12, 1);
  }
}, { output: [128, 72], graphical: true, dynamicArguments: true });

// Task 3: 72 sequential steps, three numbers each. It stays on the host.
function backtrack(cost) {
  const h = cost.length;
  const w = cost[0].length;
  const seam = new Array(h);
  let x = 0;
  for (let i = 1; i < w; i++) if (cost[h - 1][i] < cost[h - 1][x]) x = i;
  seam[h - 1] = x;
  for (let y = h - 2; y >= 0; y--) {
    let best = x;
    if (x > 0 && cost[y][x - 1] < cost[y][best]) best = x - 1;
    if (x + 1 < w && cost[y][x + 1] < cost[y][best]) best = x + 1;
    x = best;
    seam[y] = x;
  }
  return seam;
}

async function carveOnce(planes) {
  const w = planes[0][0].length;

  maskedEnergy.setOutput([w, 72]);
  const e = await maskedEnergy(planes[3], planes[4]);

  step.setOutput([w]);
  const rows = [Float32Array.from(e[0])];
  for (let y = 1; y < 72; y++) rows.push(await step(e[y], rows[y - 1]));

  const seam = backtrack(rows);
  const cost = rows[71][seam[71]];

  carve.setOutput([w - 1, 72]);
  const next = [];
  for (let i = 0; i < planes.length; i++) next.push(await carve(planes[i], seam));

  return { planes: next, cost };
}

// Five planes now: r, g, b, luminance — and the mask, which lives in image
// space and so has to reflow with everything else.
let planes = [];
for (let c = 0; c < 4; c++) planes.push(await channel(photo, c));
planes.push(faceMask);

let protectedBefore = 0;
for (let y = 0; y < 72; y++) for (let x = 0; x < 128; x++) protectedBefore += faceMask[y][x];

const costs = [];
await paint(planes[0], planes[1], planes[2], planes[0][0].length);
render(paint.canvas);

for (let k = 0; k < 32; k++) {
  const out = await carveOnce(planes);
  planes = out.planes;
  costs.push(out.cost);
  await paint(planes[0], planes[1], planes[2], planes[0][0].length);
  render(paint.canvas);
}

console.log('carved down to', planes[0][0].length, 'columns');
console.log('protected pixels before:', protectedBefore);

// TODO 2: count the 1s left in the carved mask, planes[4], and log it.
// TODO 3: plot unmaskedCosts and costs on the same axes.
```

---

Interactive version: https://gpu.rocks/learn/seam-carving-a23a0d9b/6

[Previous task](https://gpu.rocks/learn/seam-carving-a23a0d9b/5.md)