Deterministic Playwright Tests: The Foundation of Your QA Agent

Before any AI writes a test, you must prove that your project can run one. Deterministic Playwright tests give the same result every time for the same code, so they are a reliable foundation. In this lesson you will create a tiny login page, configure Playwright and watch a real test pass.

---

💡 Why Deterministic Playwright Tests Come First

If an AI-generated test fails later, you need to know why. With a proven foundation, you can rule out your setup.

---

🔍 Create a Safe Demo Application

This small server is the only site that this project tests. It runs on your own machine and uses a made-up account. Create demo/server.ts.

import { createServer } from 'node:http';

// A tiny login page that exists only for this project.
// The account below is synthetic and works nowhere else.
const page = `<!doctype html>
<html lang="en">
<head><meta charset="utf-8"><title>Demo Login</title></head>
<body>
  <main id="app">
    <h1>Sign in</h1>
    <form id="login" novalidate>
      <label for="email">Email</label>
      <input id="email" type="email" autocomplete="off">
      <label for="password">Password</label>
      <input id="password" type="password" autocomplete="off">
      <button type="submit">Sign in</button>
    </form>
    <p id="message" role="alert"></p>
  </main>
  <script>
    document.getElementById('login').addEventListener('submit', (event) => {
      event.preventDefault();
      const email = document.getElementById('email').value.trim();
      const password = document.getElementById('password').value;
      const message = document.getElementById('message');
      if (!email || !password) {
        message.textContent = 'Email and password are required';
      } else if (email === 'demo@example.test' && password === 'Demo-Pass-123') {
        document.getElementById('app').innerHTML = '<h1>Dashboard</h1><p>Welcome, demo user</p>';
      } else {
        message.textContent = 'Invalid email or password';
      }
    });
  </script>
</body>
</html>`;

createServer((request, response) => {
  if (request.url?.startsWith('/login')) {
    response.writeHead(200, { 'Content-Type': 'text/html; charset=utf-8' });
    response.end(page);
    return;
  }
  response.writeHead(404, { 'Content-Type': 'text/plain' });
  response.end('Not found');
}).listen(4321, '127.0.0.1', () => {
  console.log('Demo app running at http://127.0.0.1:4321/login');
});

---

🧩 Configure Playwright

Create playwright.config.ts.

import 'dotenv/config';
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  testMatch: '**/*.spec.ts',
  outputDir: 'artifacts/test-results',
  timeout: 30_000,
  retries: 0,
  reporter: [
    ['html', { open: 'never' }],
    ['list'],
    ['json', { outputFile: 'artifacts/results.json' }],
  ],
  use: {
    baseURL: process.env.BASE_URL,
    headless: process.env.HEADLESS !== 'false',
    screenshot: 'only-on-failure',
    trace: 'retain-on-failure',
  },
  webServer: {
    command: 'npm run demo',
    url: 'http://127.0.0.1:4321/login',
    reuseExistingServer: !process.env.CI,
  },
});

Setting	Why it matters
retries: 0	Real failures stay visible
screenshot and trace	Failures keep their evidence
JSON reporter	The executor can read results in code
webServer	Starts the demo app for you
testMatch	Keeps unit tests out of browser runs

---

🛠️ Write a Hand-Authored Test

Create tests/smoke.spec.ts.

import { test, expect } from '@playwright/test';

test('login page is available', async ({ page }) => {
  await page.goto('/login');
  await expect(page).toHaveURL(/login/);
  await expect(page.getByRole('heading', { name: 'Sign in' })).toBeVisible();
});

Run it.

npx playwright test

You should see output like this.

Running 1 test using 1 worker
  ok 1 tests/smoke.spec.ts:3:1 › login page is available

  1 passed

You can open the report with npm run test:report. If Playwright is new to you, review Install Playwright and Playwright reporters.

---

📌 Prove That Failures Leave Evidence

Change 'Sign in' to 'Welcome' in the test, and run it again. It should fail. Now check that artifacts/test-results/ contains a trace.zip and a screenshot. Change the test back when you are done.

---

🔑 Acceptance Checks for This Phase

• A hand-written test passes.
• A failing test produces a trace and a screenshot.
• The test uses a role-based locator.
• No credentials or production data are in Git.

---

🌍 Real-Life Example: The Blamed Model

A tester connects a model on day one. Its first generated test fails, so they spend a day rewriting prompts. The real problem was a wrong baseURL. A single hand-written test would have shown that immediately.

---

🧪 Try It Yourself

1. Create the three files.
2. Run the smoke test and open the report.
3. Force a failure and find the trace.
4. Open the trace with npx playwright show-trace and the path to trace.zip.
5. Open http://127.0.0.1:4321/login after npm run demo, and sign in by hand.

---

📋 Ready-to-Copy Prompt

You are a Playwright expert helping me debug a test setup.

My playwright.config.ts:
[paste it]

The command I ran and its exact output:
[paste output with no secrets]

Find the most likely cause, and tell me how to confirm it.
Do not rewrite the whole configuration. Do not assume anything that is
not shown in the output.

---

⚠️ Common Mistakes

• Connecting an AI model before any test has passed
• Turning on retries, which can hide real problems
• Saving no evidence when tests fail
• Using brittle CSS selectors in the first test
• Testing a shared website instead of a safe target

---

🎯 Key Takeaways

• Prove the test setup before adding AI.
• A local demo is a safe, repeatable target.
• Failures must leave traces and screenshots.
• A person-written test sets the quality bar.

---

🎤 Interview Questions

1. Why start with deterministic tests in an AI project?

Model answer: They prove that the environment works and give a reliable baseline. When something later fails, I can separate setup problems from model problems.

2. Why set retries to zero while building this project?

Model answer: Retries can hide flakiness. I want every failure to be visible so that it can be investigated and correctly classified.

3. What is a Playwright trace?

Model answer: It is a recording of a test run, including actions, snapshots, console messages and network activity. It is strong evidence for diagnosing a failure.

4. Why use role-based locators?

Model answer: They reflect how users and assistive technology see the page. They are usually more stable than CSS paths, and they encourage accessible markup.

5. How do you avoid testing a system you should not touch?

Model answer: I use a local demo or a named test environment, confirm that I am authorised, and enforce an allowlist in code so that the tools cannot reach anything else.

---

📝 Assignment

Add a second hand-written test that signs in with the demo account and checks for the Dashboard heading. Read the credentials from process.env. Then force it to fail and describe what the trace shows.

---

❓ FAQ

Why not test a public practice site?

You could, if its rules allow automation. A local page is faster and is the safest choice for an automated agent.

The test cannot find my browser. What should I do?

Run npx playwright install chromium again from the project folder.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Guardrails: Contracts, Context and Safety Limits for QA.