Visual Regression Testing in Playwright Python (Screenshot Comparison)

Functional assertions check text, values and states, but they cannot see that a button turned the wrong colour or a layout broke. Visual regression testing compares screenshots with approved baseline images and fails when pixels change. Playwright for TypeScript has this built in; Playwright for Python does not, so in this lesson you will build your own screenshot comparison fixture with Pillow and pixelmatch, try a ready-made pytest plugin, and use the built-in ARIA snapshot assertion to check page structure.

---

👁️ What Is Visual Regression Testing?

Imagine a CSS change that makes the “Submit” button white on a white background. Every functional test still passes: the button exists, it is enabled, its text is “Submit” and clicking it works. A user, however, cannot see it.

Visual regression testing catches this kind of bug by comparing pictures:

1. Baseline: the first time, the test takes a screenshot and saves it as the approved image.
2. Comparison: on later runs, it takes a new screenshot and compares it pixel by pixel with the baseline.
3. Diff: if too many pixels differ, the test fails and saves the new image and a diff image that highlights the changes.
4. Review: a person looks at the diff. If the change is a bug, fix the app. If it is intended (a redesign), update the baseline.

Python has no built-in to_have_screenshot

In Playwright for TypeScript, the test runner provides expect(page).toHaveScreenshot(). In Playwright for Python 1.64, there is no to_have_screenshot() assertion: expect() for pages and locators has no screenshot comparison. You have three options:

Option	Good for
Your own fixture with Pillow + pixelmatch	Full control, no extra plugin, a great way to learn how it works
A pytest plugin such as pytest-playwright-visual-snapshot	Ready-made baseline handling and options
A cloud service (Percy, Applitools)	Teams that want a web dashboard for reviewing changes

This lesson covers the first two in detail. You will also meet to_match_aria_snapshot(), a built-in Python assertion that compares the structure of the page instead of its pixels.

---

🧰 Building an assert_snapshot Fixture

Install the two libraries the fixture uses:

⌨️ Terminal

pip install pillow pixelmatch

• Pillow (PIL) opens, compares and saves images.
• pixelmatch is a Python port of the pixel comparison library used by many visual testing tools. It counts the differing pixels, ignores anti-aliasing differences at text edges, and draws a diff image.

Put the fixture in the project’s root conftest.py, so every test can use it:

📄 conftest.py

import io
import sys
from pathlib import Path

import pytest
from PIL import Image
from pixelmatch.contrib.PIL import pixelmatch

SNAPSHOT_DIR = Path("__snapshots__")          # approved baselines (commit these to Git)
RESULTS_DIR = Path("snapshot-results")        # actual and diff images of failures


def pytest_addoption(parser):
    parser.addoption(
        "--update-snapshots",
        action="store_true",
        help="Save the current screenshots as the new baselines",
    )


@pytest.fixture
def assert_snapshot(request, browser_name):
    update = request.config.getoption("--update-snapshots")

    def check(image: bytes, name: str, threshold: float = 0.1, max_diff_ratio: float = 0.0):
        __tracebackhide__ = True                # report failures at the test's line, not in here

        # One baseline per browser and operating system: rendering differs between them
        file_name = f"{name}-{browser_name}-{sys.platform}.png"
        baseline = SNAPSHOT_DIR / file_name

        # First run, or an explicit update: save the baseline
        if update or not baseline.exists():
            baseline.parent.mkdir(parents=True, exist_ok=True)
            baseline.write_bytes(image)
            if not update:
                pytest.fail(f"No baseline yet. Saved {baseline.as_posix()}: review it, then run again.")
            return

        actual = Image.open(io.BytesIO(image))
        expected = Image.open(baseline)
        if actual.size != expected.size:
            pytest.fail(f"Size changed: baseline {expected.size}, now {actual.size}")

        # Compare: returns the number of pixels that differ and draws the diff image
        diff = Image.new("RGBA", actual.size)
        different = pixelmatch(actual, expected, diff, threshold=threshold)
        ratio = different / (actual.width * actual.height)

        if ratio > max_diff_ratio:
            out = RESULTS_DIR / name
            out.mkdir(parents=True, exist_ok=True)
            actual.save(out / f"actual-{file_name}")
            expected.save(out / f"expected-{file_name}")
            diff.save(out / f"diff-{file_name}")
            pytest.fail(
                f"{different} pixels ({ratio:.2%}) differ from {baseline.as_posix()}. "
                f"See {out.as_posix()}/"
            )

    return check

Line by line:

• pytest_addoption() adds a --update-snapshots command-line flag (you learned this hook in Module 3).
• The fixture returns the inner function check(), so a test calls assert_snapshot(image_bytes, "name").
• __tracebackhide__ = True is a pytest feature: it hides this helper from failure tracebacks, so the report points at the line in your test.
• browser_name is a pytest-playwright fixture (“chromium”, “firefox” or “webkit”). Together with sys.platform (“win32”, “linux”, “darwin”) it makes the file name unique, because fonts and anti-aliasing render differently on each browser and operating system.
• First run: there is no baseline, so the image is saved and the test fails on purpose. A missing baseline must never pass silently; a person should look at it first.
• Size check: pixelmatch needs two images of the same size. A size change usually means the layout changed, so report it clearly.
• pixelmatch(actual, expected, diff, threshold=0.1) compares colour by colour. threshold (0 to 1) is how different a pixel’s colour must be to count as changed: smaller is stricter.
• max_diff_ratio is how many pixels may differ (0.0 = none, 0.01 = 1%). Raise it a little if tiny rendering noise makes a test flaky.
• On failure, the expected, actual and diff images are saved side by side for review.

---

✅ A First Visual Test

The test takes a screenshot of the web table on the practice page and compares it with the baseline:

📄 tests/test_visual.py

import pytest
from playwright.sync_api import Page


@pytest.fixture
def practice(page: Page) -> Page:
    page.goto("https://sqatools.in/automation-practice-page/")
    return page


def test_web_table_looks_right(practice: Page, assert_snapshot):
    table = practice.locator("table")
    assert_snapshot(table.screenshot(animations="disabled"), "web-table")

• An element screenshot is much more stable than a full page: the practice page also contains ads and comments that change, but the table does not.
• animations="disabled" stops CSS animations and transitions, so a moving element cannot cause a difference.

Run 1: no baseline yet.

⌨️ Terminal

pytest -q tests

🖥️ Output

F                                                                        [100%]
================================== FAILURES ===================================
____________________ test_web_table_looks_right[chromium] _____________________

practice = <Page url='https://sqatools.in/automation-practice-page/'>
assert_snapshot = <function assert_snapshot.<locals>.check at 0x00000218205DD380>

    def test_web_table_looks_right(practice: Page, assert_snapshot):
        table = practice.locator("table")
>       assert_snapshot(table.screenshot(animations="disabled"), "web-table")
E       Failed: No baseline yet. Saved __snapshots__/web-table-chromium-win32.png: review it, then run again.

tests\test_visual.py:13: Failed
=========================== short test summary info ===========================
FAILED tests/test_visual.py::test_web_table_looks_right[chromium] - Failed: N...
1 failed, 1 warning in 2.91s

Open __snapshots__/web-table-chromium-win32.png, check that it looks right, and commit it to Git.

Run 2: the comparison passes.

⌨️ Terminal

pytest -q tests

🖥️ Output

.                                                                        [100%]
1 passed, 1 warning in 2.73s

---

🔴 Catching a Visual Bug

Now simulate a developer’s CSS change that turns the table headers red. page.add_style_tag() injects a stylesheet into the page, just like a new CSS rule in the application would:

📄 demos/test_regression.py

import pytest
from playwright.sync_api import Page


@pytest.fixture
def practice(page: Page) -> Page:
    page.goto("https://sqatools.in/automation-practice-page/")
    return page


def test_table_after_css_change(practice: Page, assert_snapshot):
    practice.add_style_tag(content="th { color: red; }")      # the "bug"
    table = practice.locator("table")
    assert_snapshot(table.screenshot(animations="disabled"), "web-table")

⌨️ Terminal

pytest -q demos

🖥️ Output

F                                                                        [100%]
================================== FAILURES ===================================
____________________ test_table_after_css_change[chromium] ____________________

practice = <Page url='https://sqatools.in/automation-practice-page/'>
assert_snapshot = <function assert_snapshot.<locals>.check at 0x00000263C481D640>

    def test_table_after_css_change(practice: Page, assert_snapshot):
        practice.add_style_tag(content="th { color: red; }")      # the "bug"
        table = practice.locator("table")
>       assert_snapshot(table.screenshot(animations="disabled"), "web-table")
E       Failed: 412 pixels (0.32%) differ from __snapshots__/web-table-chromium-win32.png. See snapshot-results/web-table/

demos\test_regression.py:14: Failed
=========================== short test summary info ===========================
FAILED demos/test_regression.py::test_table_after_css_change[chromium] - Fail...
1 failed, 1 warning in 2.77s

Every functional check on this table would still pass: the same headers, the same cells, the same text. Only the visual comparison noticed. The files involved:

🖥️ Output

__snapshots__/web-table-chromium-win32.png
snapshot-results/web-table/actual-web-table-chromium-win32.png
snapshot-results/web-table/diff-web-table-chromium-win32.png
snapshot-results/web-table/expected-web-table-chromium-win32.png

• expected-...png is the baseline, actual-...png the new screenshot.
• diff-...png shows the unchanged pixels faded and the changed pixels in red. In this run, only the three header words (ID, Name, Role) were red, so you see at a glance that only the header row changed. Yellow pixels at the letter edges are anti-aliasing differences that pixelmatch detected and did not count.

Accepting an intended change

When a change is intended, update the baselines and commit them:

⌨️ Terminal

pytest -q tests --update-snapshots

🖥️ Output

.                                                                        [100%]
1 passed, 1 warning in 2.46s

---

🎯 Making Visual Tests Stable

Visual tests fail on any pixel change, including ones that are not bugs. These techniques keep them reliable:

Problem	Solution
Dates, times, ads, avatars, random data	mask=[locator, ...] in screenshot(), or style= to hide them
Animations, transitions, blinking cursor	animations="disabled"; caret="hide" is the default
Different fonts on Windows, macOS and Linux	Separate baselines per OS (as in the fixture), or always run visual tests in the same Docker image
Different window sizes	Keep the viewport fixed (pytest-playwright uses 1280 x 720)
Content still loading	Wait first, for example expect(locator).to_be_visible(), before the screenshot
Large pages with many moving parts	Screenshot components (a header, a card, a form) instead of the full page
Tiny rendering noise	A small max_diff_ratio, such as 0.001, or a higher threshold

For example, to mask the changing parts of a page:

image = page.screenshot(
    full_page=True,
    animations="disabled",
    mask=[page.locator(".ad-banner"), page.get_by_test_id("current-time")],
)
assert_snapshot(image, "home-page")

---

🔌 Option 2: The pytest-playwright-visual-snapshot Plugin

If you would rather not maintain your own fixture, the pytest-playwright-visual-snapshot plugin (version 0.5.1, which needs Python 3.11 or newer) provides an assert_snapshot fixture built on the same libraries:

⌨️ Terminal

pip install pytest-playwright-visual-snapshot

📄 plugin/pytest.ini

[pytest]

📄 plugin/test_plugin_visual.py

from playwright.sync_api import Page


def test_table_with_plugin(page: Page, assert_snapshot):
    page.goto("https://sqatools.in/automation-practice-page/")
    assert_snapshot(page.locator("table"))

• Pass a page or a locator; the plugin takes the screenshot itself, with animations disabled.
• It also accepts threshold=, name= and mask_elements= (a list of CSS selectors).
• Command-line options: --update-snapshots, --ignore-size-diff and --disable-visual-snapshots. Settings such as the snapshot folder and default masks can go in pytest.ini.
• The empty pytest.ini here only makes this folder its own pytest project for the demo. In a real project, use either your own fixture or the plugin: both define --update-snapshots and assert_snapshot, so they cannot be loaded together.

The first run creates the baseline and fails; the second run compares (... marks lines removed from the long traceback):

🖥️ Output

# First run
2026-11-01T10:35:09.415283Z [info     ] structlog output capture disabled, no output directory specified [structlog_config.pytest]
.E                                                                       [100%]
=================================== ERRORS ====================================
____________ ERROR at teardown of test_table_with_plugin[chromium] ____________
...
E           Failed: [playwright-visual-snapshot] New snapshot(s) created. Please review images. C:\pwv\playwright-python-visual-testing\plugin\__snapshots__\test_plugin_visual\test_table_with_plugin\test_table_with_plugin[chromium][win32].png
...
================================= slow tests ==================================
[slow] 2.53s test_plugin_visual.py::test_table_with_plugin[chromium]
=========================== short test summary info ===========================
ERROR test_plugin_visual.py::test_table_with_plugin[chromium] - Failed: [play...
1 passed, 1 warning, 1 error in 3.09s

# Second run
.                                                                        [100%]
================================= slow tests ==================================
[slow] 2.44s test_plugin_visual.py::test_table_with_plugin[chromium]
1 passed, 1 warning in 2.97s

The baseline file it created:

🖥️ Output

plugin/__snapshots__/test_plugin_visual/test_table_with_plugin/test_table_with_plugin[chromium][win32].png

What to notice in this real output:

• The plugin reports a new baseline or a mismatch at the end of the test, from a fixture finalizer. pytest therefore shows the test as passed plus an error at teardown, and the run still fails.
• The structlog log line and the “slow tests” section come from structlog-config, a dependency the plugin installs, which registers its own pytest plugin.
• Baselines go to __snapshots__/<test file>/<test name>/<test name>[browser][os].png, so they are already separated per browser and operating system.

Cloud visual testing services

Percy (BrowserStack) and Applitools Eyes publish Python SDKs for Playwright on PyPI (percy-playwright and eyes-playwright). They upload screenshots to a web dashboard where the team reviews and approves changes, and Applitools markets its comparison as “Visual AI” that ignores insignificant differences. Both are commercial services (check their current pricing); consider them when many people need to review visual changes.

---

🌳 Structural Snapshots: to_match_aria_snapshot

Pixel comparison catches visual changes. Sometimes you want something more robust: did the structure of a component change? Playwright for Python has a built-in assertion for this. to_match_aria_snapshot() compares the element’s accessibility tree, described in YAML:

📄 aria/test_aria.py

from playwright.sync_api import Page, expect


def test_table_structure(page: Page):
    page.goto("https://sqatools.in/automation-practice-page/")
    expect(page.locator("table")).to_match_aria_snapshot("""
      - table:
        - rowgroup:
          - row "ID Name Role"
        - rowgroup:
          - row "1 Deepesh Trainer"
          - row "2 Rahul Tester"
          - row "3 Anita Developer"
    """)


def test_partial_and_regex(page: Page):
    page.goto("https://sqatools.in/automation-practice-page/")
    # Only list what matters: other options and attributes may exist
    expect(page.locator("#country")).to_match_aria_snapshot("""
      - combobox:
        - option "India"
        - option /U.*/
    """)


def test_print_a_snapshot(page: Page):
    page.goto("https://sqatools.in/automation-practice-page/")
    print("\n" + page.locator("#country").aria_snapshot())

⌨️ Terminal

pytest -q -s aria

🖥️ Output

..
- combobox:
  - option "–Select Country–" [selected]
  - option "India"
  - option "USA"
  - option "UK"
  - option "Australia"
.
3 passed, 1 warning in 6.62s

• Each line is - role "accessible name", indented to show nesting.
• The template matches partially: you can leave out children, such as the rows’ cells and the other countries above. The children you list must appear in that order.
• /U.*/ is a regular expression for the name; here it matched “USA”.
• locator.aria_snapshot() returns the snapshot as text. Print it once, check it, and paste it into your test.
• ARIA snapshots ignore colours and layout, so they do not replace pixel comparison. They are excellent for menus, tables and forms whose content and structure must not change, and they do not depend on the operating system’s fonts.

---

❓ Frequently Asked Questions

🤔 Does Playwright Python have toHaveScreenshot?

No. to_have_screenshot() exists only in the TypeScript/JavaScript test runner. In Python, compare screenshots with your own fixture (Pillow and pixelmatch), a pytest plugin, or a cloud service.

🤔 Where should I store baseline images?

In your Git repository, next to the tests, so a baseline change is reviewed in the same pull request as the code change that caused it.

🤔 Why does my visual test fail on CI but pass locally?

Usually different fonts or rendering on another operating system. Create the baselines in the same environment that compares them, typically a fixed Docker image in CI, and keep one baseline per browser and OS.

🤔 What threshold should I use?

Start with pixelmatch’s default threshold=0.1 and max_diff_ratio=0 (no differing pixels). Raise them only if you see real noise, and only as much as needed; a high tolerance hides real bugs.

🤔 Can I compare a full page?

Yes, with page.screenshot(full_page=True), but full pages contain more changing content. Mask dynamic parts, or prefer component screenshots for stable tests.

---

🏁 Summary

• Visual regression testing compares screenshots with approved baselines and shows the differences in a diff image.
• Playwright for Python has no to_have_screenshot(); build an assert_snapshot fixture with Pillow and pixelmatch, or use a plugin such as pytest-playwright-visual-snapshot.
• A missing baseline should fail; --update-snapshots accepts intended changes after review.
• Keep visual tests stable: element screenshots, masks, disabled animations, a fixed viewport, and one environment for baselines.
• to_match_aria_snapshot() is the built-in way to check a component’s structure, independent of fonts and colours.

You have finished Module 5. Test yourself with the Module 5 mock interview, then continue to Module 6, where you will master Playwright’s pytest fixtures: browser, context and page scopes, and your own fixtures.