Retries and Flaky Tests in Playwright Python with pytest-rerunfailures

A flaky test passes on one run and fails on the next, with no change in the code. Flaky tests destroy trust in a test suite: people stop believing red builds. The Playwright TypeScript runner has retries built in; in Python, the pytest-rerunfailures plugin provides them. In this lesson you will learn what makes Playwright tests flaky and how to fix the cause, then use retries correctly: –reruns, delays, the flaky marker, rerunning only specific errors, failing the build on flaky passes, artifacts per attempt, and finding flaky tests with pytest-repeat.

---

🎲 What Is a Flaky Test?

A flaky test gives different results on different runs of the same code: green on your laptop, red in CI, green again when someone clicks “re-run”. It is one of the most expensive problems in test automation:

• Real bugs hide among the “probably flaky again” failures.
• Teams waste hours re-running pipelines and investigating ghosts.
• People stop trusting the suite, and a suite nobody trusts is worth nothing.

A real example from this course: while Lesson 42 was being prepared, about one in six navigations to the public SauceDemo site stalled for more than 30 seconds, and the same unchanged tests failed with Page.goto: Timeout 30000ms exceeded in some runs and passed in others.

---

🔍 Common Causes and Their Real Fixes

Retries hide symptoms. Before you add them, look for the cause:

Cause	Symptom	Real fix
Checking too early	Passes locally, fails on slow CI	Use expect() assertions, never time.sleep() or one-shot reads like inner_text() (Module 5)
Shared state between tests	Fails only in a certain order or in parallel	Fresh context per test (the default), unique test data, no module-level globals
Animations and transitions	Clicks land on a moving element	Let actionability checks work; disable animations for screenshots
Dependence on live third-party services	Random timeouts	Mock the network (Module 8), or use a stable test environment
Dates, times and random data	Fails around midnight or on certain values	Fix the time zone with timezone_id, control test data
Fragile locators	Fails after small UI changes or with A/B tests	Role-based locators and test ids (Module 4)
Too-short timeouts	Fails under load	Timeouts based on real requirements, not guesses

---

🔁 Retries with pytest-rerunfailures

Install the plugin:

⌨️ Terminal

pip install pytest-rerunfailures

This lesson uses version 16.7. To make the demonstrations repeatable, the test below simulates a flaky server. It shows a search result after a delay: 6 seconds on the first attempt (longer than expect()‘s 5-second timeout) and 1 second on later attempts. pytest-rerunfailures numbers the attempts in request.node.execution_count (1, 2, 3, …).

📄 flaky/test_flaky.py

from playwright.sync_api import Page, expect

RESULT_PAGE = """
<p id="result">Searching...</p>
<script>
  setTimeout(() => document.getElementById('result').textContent = '3 products found', {delay});
</script>
"""


def test_search_results(page: Page, request):
    attempt = getattr(request.node, "execution_count", 1)   # 1 = first try, 2 = first retry, ...
    delay = 6_000 if attempt == 1 else 1_000                # simulated slow server on the first try
    page.set_content(RESULT_PAGE.replace("{delay}", str(delay)))

    expect(page.locator("#result")).to_have_text("3 products found")

• getattr(..., 1) returns 1 when the plugin is not active, so the test also runs without it.
• In a real suite the delay comes from the server, not from your test. The simulation only makes the behaviour predictable for this lesson.

Without retries, the first attempt fails:

⌨️ Terminal

pytest -q flaky

🖥️ Output

F                                                                        [100%]
================================== FAILURES ===================================
________________________ test_search_results[chromium] ________________________

page = <Page url='about:blank'>
request = <FixtureRequest for <Function test_search_results[chromium]>>

    def test_search_results(page: Page, request):
        attempt = getattr(request.node, "execution_count", 1)   # 1 = first try, 2 = first retry, ...
        delay = 6_000 if attempt == 1 else 1_000                # simulated slow server on the first try
        page.set_content(RESULT_PAGE.replace("{delay}", str(delay)))
    
>       expect(page.locator("#result")).to_have_text("3 products found")
E       AssertionError: Locator expected to have text '3 products found'
E       Actual value: Searching... 
E       Call log:
E         - Expect "to_have_text" locator("#result") with timeout 5000ms
E         - waiting for locator("#result")
E           14 × locator resolved to <p id="result">Searching...</p>
E              - unexpected value "Searching..."
E       
E       Aria snapshot:
E       - paragraph: Searching...

flaky\test_flaky.py:16: AssertionError
=========================== short test summary info ===========================
FAILED flaky/test_flaky.py::test_search_results[chromium] - AssertionError: L...
1 failed, 1 warning in 5.56s

With retries, --reruns 2 allows up to two more attempts for a failed test:

⌨️ Terminal

pytest -q -rR flaky --reruns 2

🖥️ Output

R.                                                                       [100%]
=========================== rerun test summary info ===========================
RERUN flaky/test_flaky.py::test_search_results[chromium]
PASSED flaky/test_flaky.py::test_search_results[chromium]
1 passed, 1 warning, 1 rerun in 6.95s

• R in the progress line means “failed, then rerun”; the final . is the passing attempt.
• The summary counts reruns separately: 1 passed, 1 rerun. The run’s exit code is 0.
• -rR adds the “rerun test summary info” section that lists which tests needed a rerun. Keep an eye on this list: every entry is a flaky test to investigate.
• Each attempt runs the test’s fixtures again: a new context and a new page.

---

⚙️ Retry Options

Option	Meaning
--reruns N	Rerun a failed test up to N times
--reruns-delay S	Wait S seconds before each rerun (for example, to let a service recover)
--only-rerun REGEX	Rerun only failures whose error matches the regex; repeatable
--rerun-except REGEX	Do not rerun failures that match the regex; repeatable
--fail-on-flaky	Fail the run with exit code 7 if any test passed only after a rerun
--max-suite-reruns N	Limit the total number of reruns in the whole run
--force-reruns N	Use N reruns for every test, ignoring markers

The same settings work in pytest.ini:

[pytest]
addopts = --reruns 2 --reruns-delay 1

pytest-rerunfailures 16.7 also has its own ini keys, such as reruns = 2, reruns_delay = 1, only_rerun and rerun_except.

Retry only infrastructure errors: –only-rerun

A retry makes sense for problems outside your application, such as a slow network. It makes no sense for a wrong text, which will be wrong again on the next attempt. Here is a real bug and a slow page in one file:

📄 filters/test_filters.py

from playwright.sync_api import Page, expect

SLOW_BUTTON = """
<script>
  setTimeout(() => document.body.innerHTML = '<button>Checkout</button>', {delay});
</script>
"""


def test_wrong_heading(page: Page):
    page.set_content("<h1>Cart</h1>")
    expect(page.locator("h1")).to_have_text("Basket", timeout=1_000)   # a real bug: wrong text


def test_slow_button(page: Page, request):
    attempt = getattr(request.node, "execution_count", 1)
    page.set_content(SLOW_BUTTON.replace("{delay}", "3000" if attempt == 1 else "200"))
    page.get_by_role("button", name="Checkout").click(timeout=2_000)       # an action timeout

⌨️ Terminal

pytest -q filters --reruns 2 --only-rerun TimeoutError

🖥️ Output

FR.                                                                      [100%]
================================== FAILURES ===================================
________________________ test_wrong_heading[chromium] _________________________

page = <Page url='about:blank'>

    def test_wrong_heading(page: Page):
        page.set_content("<h1>Cart</h1>")
>       expect(page.locator("h1")).to_have_text("Basket", timeout=1_000)   # a real bug: wrong text
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AssertionError: Locator expected to have text 'Basket'
E       Actual value: Cart 
E       Call log:
E         - Expect "to_have_text" locator("h1") with timeout 1000ms
E         - waiting for locator("h1")
E           11 × locator resolved to <h1>Cart</h1>
E              - unexpected value "Cart"
E       
E       Aria snapshot:
E       - heading "Cart" [level=1]

filters\test_filters.py:12: AssertionError
=========================== short test summary info ===========================
FAILED filters/test_filters.py::test_wrong_heading[chromium] - AssertionError...
1 failed, 1 passed, 1 warning, 1 rerun in 4.07s

• test_slow_button failed with Playwright’s TimeoutError on the first attempt, matched TimeoutError, was rerun and passed.
• test_wrong_heading failed without a rerun. A failed expect() raises an AssertionError, even when it waited for its timeout, so it does not match TimeoutError.

Failing the build on flaky passes: –fail-on-flaky

Some teams want retries to keep the pipeline moving, but still want to know about flakiness. --fail-on-flaky lets the tests pass on retry but ends the run with exit code 7:

⌨️ Terminal

pytest -q flaky --reruns 2 --fail-on-flaky

🖥️ Output

R.                                                                       [100%]
1 passed, 1 warning, 1 rerun in 6.96s

The summary still says 1 passed, 1 rerun, but pytest exited with code 7 (in our run, the verification script recorded exit 7), so CI marks the job as failed and someone has to look at the flaky test.

---

🏷️ The flaky Marker: Retries for Specific Tests

Instead of retrying the whole suite, mark only the tests you know are affected:

📄 marker/test_marker.py

import pytest
from playwright.sync_api import Page, expect

RESULT_PAGE = """
<p id="result">Searching...</p>
<script>
  setTimeout(() => document.getElementById('result').textContent = 'done', {delay});
</script>
"""


@pytest.mark.flaky(reruns=3, reruns_delay=1, only_rerun=["AssertionError"])
def test_known_slow_search(page: Page, request):
    attempt = request.node.execution_count
    print(f"\nattempt {attempt}")
    page.set_content(RESULT_PAGE.replace("{delay}", "4000" if attempt < 3 else "100"))
    expect(page.locator("#result")).to_have_text("done", timeout=2_000)

⌨️ Terminal

pytest -q -s marker

🖥️ Output

attempt 1
R
attempt 2
R
attempt 3
.
1 passed, 1 warning, 2 rerun in 6.89s

• reruns=3 allows three reruns for this test only, without any command-line option.
• reruns_delay=1 waits one second between attempts.
• only_rerun= (and rerun_except=) work in the marker too. Here the test’s failures are AssertionErrors from expect(), so they are listed explicitly.
• The test failed twice (each R) and passed on attempt 3; the summary counts 2 rerun.
• When both the marker and --reruns are given, the marker wins by default (--reruns-mode strict).

---

📸 Artifacts for Each Attempt

When you combine retries with pytest-playwright’s artifacts, each failed attempt keeps its own evidence. Run the simulated flaky test with one rerun and screenshots on failure:

⌨️ Terminal

pytest -q flaky --reruns 1 --screenshot only-on-failure

🖥️ Output

R.                                                                       [100%]
1 passed, 1 warning, 1 rerun in 7.00s

The output folder now contains:

🖥️ Output

test-results/flaky-test-flaky-py-test-search-results-chromium/test-failed-1.png

• The screenshot of the first, failed attempt is kept, which is exactly the evidence you need to understand the flaky failure.
• The retry passed, so it kept nothing. A failed retry would have its own folder ending in -retry1, as you saw in Module 5.
• With --tracing retain-on-failure, you get a trace of the failed attempt too: the best way to see why it was slow.

---

🔂 Finding Flaky Tests: pytest-repeat

To check whether a test is stable before it causes trouble in CI, run it many times. The pytest-repeat plugin (version 0.9.4 here) adds --count:

⌨️ Terminal

pip install pytest-repeat

📄 repeat/test_repeat.py

from playwright.sync_api import Page, expect


def test_practice_page_title(page: Page):
    page.goto("https://sqatools.in/automation-practice-page/")
    expect(page).to_have_title("Automation Practice Page - SQA Tools")

⌨️ Terminal

pytest repeat --count 3 -v

🖥️ Output

============================= test session starts =============================
platform win32 -- Python 3.15.1, pytest-9.2.0, pluggy-1.7.0 -- C:\pwv\_venv\Scripts\python.exe
...
plugins: html-4.2.0, metadata-3.1.1, playwright-0.10.0, repeat-0.9.4, rerunfailures-16.7, xdist-3.8.0
collecting ... collected 3 items

repeat/test_repeat.py::test_practice_page_title[chromium-1-3] PASSED     [ 33%]
repeat/test_repeat.py::test_practice_page_title[chromium-2-3] PASSED     [ 66%]
repeat/test_repeat.py::test_practice_page_title[chromium-3-3] PASSED     [100%]

======================== 3 passed, 1 warning in 6.59s =========================

• Each repetition gets its own test id (1-3, 2-3, 3-3) and a fresh context.
• For a real stability check, use a higher count, such as --count 50, and add -x to stop at the first failure. Combine it with pytest-xdist (Module 12) to finish faster.
• Do not combine --count with --reruns for this purpose: retries would hide exactly the failures you are looking for.

---

✅ A Sensible Retry Policy

1. Locally: no retries. You want to see every failure while you work.
2. In CI: a small number, such as --reruns 1 or --reruns 2, ideally with --only-rerun TimeoutError for infrastructure problems.
3. Always keep evidence: --tracing retain-on-failure and --screenshot only-on-failure.
4. Track every rerun. Use -rR or the report to list flaky tests, and fix them like bugs; consider --fail-on-flaky once the suite is mostly stable.
5. Never use retries to hide a real failure. A test that fails 1 time in 10 because of the application is a bug report, not noise.

---

❓ Frequently Asked Questions

🤔 Does Playwright Python have built-in retries like playwright.config.ts?

No. The TypeScript test runner has a retries setting. In Python, retries come from the pytest-rerunfailures plugin: --reruns N or the flaky marker.

🤔 Are fixtures run again on a rerun?

Yes. Each attempt is a complete run of the test, with its function-scoped fixtures (including page and context) set up and torn down again. Session-scoped fixtures, such as the browser, are reused.

🤔 Why is my expect() failure not retried with –only-rerun TimeoutError?

A failed expect() raises AssertionError, not TimeoutError. Only actions and navigations (click, fill, goto) raise Playwright’s TimeoutError. Use --only-rerun AssertionError if you really want to retry assertion failures.

🤔 How do I know which tests are flaky?

Look for R in the progress line and use -rR to list rerun tests in the summary. Run suspicious tests many times with pytest-repeat’s --count.

🤔 Can I use –reruns together with pytest-xdist?

Yes. Each worker reruns its own failed tests. Parallel runs are covered in Module 12.

---

🏁 Summary

• Flaky tests pass and fail on the same code; fix their causes first: waits, shared state, live dependencies, data and locators.
• pytest-rerunfailures adds retries: --reruns N, --reruns-delay, the flaky marker, and filters with --only-rerun and --rerun-except.
• Failed expect() checks raise AssertionError; action and navigation timeouts raise TimeoutError.
• --fail-on-flaky exits with code 7 when a test only passed on retry; -rR lists rerun tests.
• Failed attempts keep their screenshots and traces; pytest-repeat’s --count exposes flakiness before CI does.

You have finished Module 6. Test yourself with the Module 6 mock interview, then continue to Module 7: test data, configuration and cross-browser testing at scale.