Flaky Test Diagnosis with AI: Triage CI Failures Faster

A test that passes and fails with no code change slowly destroys trust in your suite. AI can help you sort failures and spot patterns, but it can also suggest quick fixes that hide real bugs. In this lesson you will practise flaky test diagnosis with AI using evidence, and fix the real cause.

---

💡 What Is a Flaky Test?

Common cause	Clue	Real fix
Timing	Fails when the page is slow	Use web-first assertions
Shared data	Fails in parallel only	Give each test its own data
Test order	Fails after another test	Remove dependencies
Unstable environment	Many unrelated failures	Fix or report the environment
Unstable locator	Matches different elements	Use a more specific locator
Product race condition	Real users may see it too	Report it as a defect

---

🔍 Triage First

Category	Next step
Product defect	Report it, with evidence
Test defect	Fix the test
Test data	Repair or isolate the data
Environment	Report it to the owner
Flaky, cause unknown	Investigate and track it

---

🧩 Collect Evidence

Configure Playwright to keep useful evidence when something goes wrong:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
});

To check whether a test is unstable, run it many times:

npx playwright test tests/example.spec.ts --repeat-each=20

Inspect failures with the Trace Viewer. A test that fails and then passes on a retry is marked flaky in the Playwright report.

---

🛠️ Flaky Test Diagnosis with AI

Good input for AI includes the exact error, the test code, how often it fails and what changed recently. Remove secrets before sharing logs.

---

🌍 Real-Life Example: A Fixed Wait

On our practice page, the message takes about three seconds to appear. This test waits for exactly that long, so it is a race. Sometimes the text is ready, and sometimes it is not.

import { test, expect } from '@playwright/test';

test('flaky version', async ({ page }) => {
  await page.goto('https://sqatools.in/automation-practice-page/');
  await page.getByRole('button', { name: 'Load Success' }).click();
  await page.waitForTimeout(3000);

  const message = await page.locator('#loader').textContent();
  expect(message).toBe('Data loaded successfully!');
});

The textContent call reads the value once, and that plain expect never retries. Here is the better version:

import { test, expect } from '@playwright/test';

test('stable version', async ({ page }) => {
  await page.goto('https://sqatools.in/automation-practice-page/');
  await page.getByRole('button', { name: 'Load Success' }).click();
  await expect(page.locator('#loader')).toHaveText('Data loaded successfully!');
});

The new assertion keeps checking until the text appears or the timeout ends. Learn more in Playwright Waits Explained.

---

🔑 Retries Are Not a Fix

If you must temporarily remove a flaky test from the main run, log an owner and a date to fix it. Otherwise, quarantined tests are easily forgotten.

---

🧪 Try It Yourself

1. Put the flaky and stable tests in your project.
2. Run both with --repeat-each=20.
3. Open a report, and compare results.
4. Run the prompt below on any failure you see.
5. Confirm the suggested cause with the trace before making a change.

---

📋 Ready-to-Copy Prompt

You are a senior Playwright engineer helping me triage a CI failure.

Test code:
[paste test]

Exact error:
[paste error]

Failure pattern: [how often, which browser, parallel or not]
Recent changes: [changes]
Trace or log notes: [sanitised details]

Classify the failure as Product Defect, Test Defect, Test Data,
Environment or Unknown.

Rules:
- Quote the evidence for your classification.
- Give your confidence and at least one alternative explanation.
- Suggest how to confirm the cause.
- Do not suggest more retries, longer waits or weaker assertions as a fix.

---

⚠️ Common Mistakes

• Rerunning until it goes green, and moving on
• Adding waitForTimeout to hide timing problems
• Assuming that every failure is flaky
• Letting a failing test weaken its own assertion
• Quarantining tests with no owner

---

🎯 Key Takeaways

• A flaky test changes result with no relevant change.
• Triage failures before you try to fix them.
• Use evidence from traces, logs and repeated runs.
• Retries should not replace real fixes.

---

🎤 Interview Questions

1. What is a flaky test, and why is it a problem?

Model answer: It is a test that gives inconsistent results without a relevant code change. It wastes time and reduces trust, so teams may begin ignoring failures, including real ones.

2. How do you investigate a flaky Playwright test?

Model answer: I run it repeatedly, then compare traces, screenshots, timing and parallel behaviour. I check for data conflicts and environment problems, then fix the root cause.

3. How do you decide if a failure is a product bug or test problem?

Model answer: I compare the actual result with the requirement, reproduce it manually if possible and review the trace and logs. If users would see the failure, it may be a product defect, even if it is intermittent.

4. Are retries good or bad?

Model answer: They are useful for handling occasional environmental problems and for capturing evidence. They become harmful when used to hide flakiness, so I still track and investigate failures that pass on retry.

5. How can AI help with failure triage?

Model answer: It can summarise logs, group similar failures and suggest possible causes. I give it sanitised evidence and require it to cite that evidence, then I confirm the cause before changing anything.

---

📝 Assignment

Write a test with a deliberate timing problem, then prove that it is unstable by repeating it. Fix it without adding a fixed wait. Record the error, cause, fix and evidence as a short triage note.

---

❓ FAQ

How many repeats prove that a test is stable?

No number gives complete proof. More repeats increase confidence, especially if you also run under realistic load and in CI.

Should flaky tests block a release?

It depends on what they cover and your team’s rules. A flaky test in a critical area needs prompt attention, as it may hide a real problem.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Review AI-Generated Automation Tests: Catch Weak Playwright Code.