Docs / Testing and review

Visual regression testing in CI

Screenery's visual regression testing compares every screenshot with a baseline before it publishes anything. This page calls that check the visual-diff gate. A visual regression fails the job, and nothing is pushed. The gate is optional: it runs only when you give it a baseline.

What the gate does

A baseline is a folder of screenshots you have approved, committed to your repository. The gate compares each screenshot your tests just wrote with the file of the same name in that folder, pixel by pixel. If anything changed, the job fails before Screenery is contacted.

A failing gate pushes nothing. No version is stored, and no channel moves, so a regression never reaches @latest or your README.

If you do not set a baseline, there is no gate, and every push publishes as usual.

Already doing screenshot testing with Playwright's toHaveScreenshot? You can keep that as your gate and have Screenery publish the goldens it accepted. See Playwright goldens and Argos folders.

To diff one pair of images by hand, use the free screenshot comparison tool. It runs in your browser. Related guide: Playwright visual regression testing.

Create the baseline

Run your tests once, then copy their screenshots into a new baseline folder:

npx screenery diff ./test-results --baseline ./screenery-baseline --update

If ./screenery-baseline does not exist, --update creates it from ./test-results. Commit the folder. Keep it outside the folder you push: a baseline inside the pushed folder would be published as screenshots, so the push refuses it.

With the Action

Point diff-baseline at the committed folder:

permissions:
  id-token: write
  contents: read
steps:
  - uses: actions/checkout@v4
  # … your Playwright step writes test-results/ …
  - uses: screenery/push@v1
    with:
      project: acme/dashboard
      path: test-results
      diff-baseline: ./screenery-baseline

screenery/push@v1 runs CLI 0.4.0, and everything on this page works with it. The Action reads screenery.config.json from your checkout, so the settings in Graded changes need no new input.

With the CLI

The gate is its own command, so you can run it anywhere, with no credential and no network:

npx screenery diff ./test-results --baseline ./screenery-baseline

It exits 1 when the gate fails and 0 when it passes. The same gate runs inside a push:

npx screenery push ./test-results --project acme/dashboard \
  --diff-baseline ./screenery-baseline --diff-threshold 0.1

The push compares first, and contacts Screenery only if the comparison passes. On a push the flags are --diff-fail-on and --diff-out.

What fails the gate

This is the default, failOn: any. To fail only on bigger changes, see Graded changes.

The log lists every screenshot that is not unchanged, and why. screenery diff also prints the share of pixels that changed. When the gate fails inside a push, the log says Nothing was pushed.

Accepting a change

When the change is what you meant, make it the new baseline and commit it:

npx screenery diff ./test-results --baseline ./screenery-baseline --update
git add screenery-baseline

--update replaces the whole baseline with the current screenshots. A screenshot you deleted from your tests leaves the baseline too.

Viewports and findings

Two or more screenshots in one folder are a family: one screen at several viewport sizes. Put the viewport last in the path, so each screen is one folder:

test-results/checkout/payment-form/mobile.png
test-results/checkout/payment-form/desktop.png
test-results/checkout/payment-form/findings.json

A findings.json beside them is a sidecar: a list of problems your checks found on that screen, such as accessibility or layout issues. It is not uploaded as a file. The gate compares it with the baseline's copy and reports each finding that is new (regressed) or gone (fixed). A new finding fails the gate, even when no pixel changed. On a push, the findings also become a card image at {family}/findings, so they have a URL like any screenshot. Naming rules are on URLs and asset names.

Comparing versions in the console

Every push is kept as a version, so you can also compare after the fact. In the console, the compare view on a public screenshot grades what changed between two versions and shows it several ways: the changes outlined, side by side, a wipe slider, flicker, onion skin, a heat map and a structure map. Unlisted and private screenshots, and videos, get a plain side-by-side view.

Agents can compare channels too, such as a pull request's @pr-12 against @latest. See Agents.

Graded changes

Each changed region is graded minor (moved up to 8 pixels, or a slight shade change), moderate (part of a region redrawn, such as edited text, or a clear recolour) or major (a colour flip such as blue to red, a replaced shape, or a size change). The same two PNGs get the same grade on every machine.

Choose which grade fails in screenery.config.json, in your repository root:

{
  "diff": {
    "failOn": "major"
  }
}

• failOn is any, minor, moderate, major or never. The default, any, fails on any changed pixel, as in What fails the gate.
• The same section holds threshold and the grading thresholds. Unknown keys are refused, so a typo cannot silently fall back to a default. A flag wins over the file for one run.
• Visual changes in the pull-request comment. When the gate passes with changes below failOn, the comment starts with them, worst first, with each one's changes image and flicker GIF. Those images count toward storage. A private project's images are uploaded but not shown. "commentImages": false turns the upload off.
• Grading without a baseline folder. Every push also grades each PNG against the version it replaces. alertAbove (a percent) warns and lists big changes at the top of the comment. failAbove fails the push before anything is published. Both are off by default. --no-change-grade skips grading for one push.
• A report after publishing. The Action's fail-on input (changed, removed, added) fails the job after the build is published, and report-out writes the report as JSON. See Inputs.

Noise, anti-aliasing and masks

Requires CLI 0.5.0. Font smoothing that lands a subpixel differently is not a change: the diff drops anti-aliased pixels. A second, stricter colour pass catches a faint recolour of a whole background, but only when it covers at least 3% of the screenshot, so dithering never counts. A change of a few pixels on a small component still counts, because the noise floor shrinks with the image.

Upgrading from an earlier CLI moves some grades once: anti-aliasing-only changes stop failing, and faint recolours that were invisible before start to show as minor tints. Set "engine": 1 in the diff section to keep the old behaviour. Every stored grade records the engine that made it.

One screenshot can loosen its own threshold, or mask a clock, an avatar or an ad, in the {name}.meta.json beside it:

{ "threshold": 0.3, "ignore": [{ "x": 0, "y": 0, "w": 320, "h": 40 }] }

Masked rectangles are filled with flat grey on both sides before comparing, so they never change. The console's compare view uses the same settings for that screenshot. See Descriptions and tags for the sidecar.

Masking by selector

Requires CLI 0.5.0. You do not have to work out pixel coordinates. The screenery/playwright helper takes the screenshot, finds every element your selectors match, and writes their boxes into the .meta.json beside the PNG as ignore rectangles:

// npm i -D screenery
import { screenshotWithIgnore } from 'screenery/playwright';

await screenshotWithIgnore(page, 'test-results/dashboard.png', {
  ignore: ['.timestamp', '[data-testid=avatar]', page.getByRole('img', { name: 'Ad' })],
  threshold: 0.3,   // optional
  fullPage: true,   // and any other page.screenshot() option
});

If you already call screenshot(page, testInfo, name) from the same module (see Which test took it), pass the same ignore and threshold options to it instead.

• The published image is not painted. Playwright's own mask option draws a solid box into the PNG, so the box is what everyone sees. ignore leaves the PNG as captured, clock and all. Only the diff skips those areas.
• Selectors or Locators. A string is a Playwright selector: CSS by default, and it reaches into open shadow roots. Every match is masked. A selector that matches nothing masks nothing and is not an error, so one test can cover pages with and without the element. Elements inside an iframe are not supported.
• Placed for you. Boxes are measured right after the capture. They follow fullPage (page coordinates rather than the scrolled viewport), clip, and the device scale factor, because rectangles are image pixels: a 100 px element on a 2x screen is a 200 px rectangle. Edges round outward, and anything past the edge of the image is clipped off.
• At most 32 rectangles. Repeats, and boxes inside another box, count once. Past 32 the call fails the test and says how many each selector matched. Select the element that holds them, such as a whole table column, rather than each cell.
• Other keys are kept. A description, tags or a capture block already in the sidecar stays. ignore is replaced on every run, never added to. threshold changes only when you pass it.

Stricter grading (engine 3)

Requires CLI 0.5.0. Opt-in. The default grader, engine 2, misses two kinds of change that engine 3 catches:

• A small colour change. A button that goes from indigo to violet on a long page is a clear change, but engine 2's faint-colour pass only counts when it covers 3% of the screenshot, and the button is under 1%. Engine 3 judges each patch of changed colour on its own: a solid patch of at least 64 pixels whose average colour moved by at least 4 ΔE counts, however small a share of the page it is. Rendering noise and dithering do not form solid patches and average out, so they still do not count.
• An element removed while the content below moves up. When a short label disappears and the text under it moves up a few pixels to fill the gap, engine 2 can call the whole area "Moved 8px up", a minor change, so failOn: moderate passes it. Engine 3 only calls a change a move when the move explains every changed pixel, both where content arrived and where it left. The vanished label has nowhere to have gone, so the area is graded as changed content (moderate or major). Rows that scroll past the edge of the screenshot do not count against a move.

Turn it on in screenery.config.json:

{
  "diff": {
    "engine": 3
  }
}

Engine 3 can fail a build that engine 2 passes, which is why it is not the default. Every stored grade records the engine that made it, and the console's compare view regrades each version with that same engine. Two optional keys tune it: colorMinRegionPx (default 64, the smallest patch that counts) and colorMinDeltaE (default 4, how far its average colour has to move). Engines 1 and 2 ignore both.

Flaky screenshots and ignored changes

A screenshot that keeps changing on @latest with nobody changing it is noise: a clock, a carousel, a font that loads late. Screenery reads that from the screenshot's own history. Its page in the console says how steady it has been over the last week, in one line such as Changed 6 times in 7 days, 4 of them the same change. Likely flaky., with the repeated changes listed and when @latest last moved it.

Fix the capture first: wait for fonts and network, or mask the moving part with an ignore rectangle as above. When a change is genuinely noise, open it in the version history and choose Ignore this change, or ignore #N only for one region of it. Every later push of that screenshot drops the ignored change before grading, so it no longer reaches the PR comment, diff.failAbove or --fail-on changed. Ignoring one region keeps the rest of the screenshot graded, so a real change in the same push still shows.

Nothing disappears silently. The push log and the stored grade say what was dropped, the PR comment counts "3 changed (1 ignored)", and the project's Ignored page lists every ignored change with who added it, why, and an unignore button. Ignoring and unignoring need the publisher role. A screenshot whose whole change was ignored needs no approval in the pull request's review, so it does not hold the screenery check either. An admin who wants a person to approve those too sets Ignored changes in review to review it on the Ignored page. Dropping happens in the CLI, so it requires CLI 0.5.0; the history and the Ignored page work for every push.

Auto-ignore is off unless an admin turns it on, on the Ignored page. With it on, a change seen a set number of times on @latest within 7 days (3 is a sensible start) is ignored automatically after the next build that moves @latest, and marked as added by automation. It never adds back a change a person unignored. A team reworking one header three times in a week looks the same as a flake, so keep the number high, or keep it off.