AI Test Executor: Run Playwright Tests in a Controlled Way

An agent that can run any command can do anything your account can do. An AI test executor has the opposite design. It can run only a short, fixed list of commands, and no model can add to that list. It also captures the results in a form that the rest of your system can read.

---

💡 Design the AI Test Executor

The list of commands is in code, so it cannot be changed at runtime. The executor also separates execution from reasoning. A model decides nothing here. This follows the rule from AI Agent Guardrails.

---

🔍 Keep Writes in One Place

Create src/tools/filesystem.tool.ts. Each file operation passes through the path guardrail.

import { mkdir, readFile, rm, writeFile } from 'node:fs/promises';
import { dirname, resolve } from 'node:path';
import { GuardrailError, isSafeGeneratedPath } from '../core/guardrails.js';
import type { GeneratedFile } from '../schemas/generation.schema.js';

export class FileSystemTool {
  constructor(private readonly root: string) {}

  private resolveGenerated(path: string): string {
    if (!isSafeGeneratedPath(path)) {
      throw new GuardrailError(`Path is not approved: ${path}`);
    }
    return resolve(this.root, path);
  }

  async writeGenerated(file: GeneratedFile): Promise<void> {
    const target = this.resolveGenerated(file.path);
    await mkdir(dirname(target), { recursive: true });
    await writeFile(target, file.content, 'utf8');
  }

  async readGenerated(path: string): Promise<string> {
    return readFile(this.resolveGenerated(path), 'utf8');
  }

  async removeGenerated(path: string): Promise<void> {
    await rm(this.resolveGenerated(path), { force: true });
  }
}

---

🧩 Build the Executor

Create src/tools/test-runner.tool.ts.

import { execFile } from 'node:child_process';
import { cp, readFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import { stripVTControlCharacters } from 'node:util';
import { GuardrailError, LIMITS } from '../core/guardrails.js';
import type { FailedTest, TestRunResult } from '../core/types.js';

const COMMANDS = {
  generatedTests: ['node_modules/@playwright/test/cli.js', 'test', 'tests/generated'],
  typecheck: ['node_modules/typescript/bin/tsc', '--noEmit'],
  lint: ['node_modules/eslint/bin/eslint.js', 'tests/generated'],
} as const;

export type CommandName = keyof typeof COMMANDS;
export type CommandResult = { exitCode: number; output: string };

const SAFE_ENV_KEYS = [
  'PATH', 'SystemRoot', 'TEMP', 'TMP', 'HOME', 'USERPROFILE', 'ComSpec',
  'PLAYWRIGHT_BROWSERS_PATH', 'CI', 'BASE_URL', 'HEADLESS',
  'TEST_USER_EMAIL', 'TEST_USER_PASSWORD',
];

type Attachment = { name: string; path?: string };
type JsonResult = { errors?: Array<{ message?: string }>; attachments?: Attachment[] };
type JsonTest = { status: string; results: JsonResult[] };
type JsonSpec = { title: string; tests: JsonTest[] };
type JsonSuite = { specs?: JsonSpec[]; suites?: JsonSuite[] };
export type JsonReport = {
  suites: JsonSuite[];
  stats: { expected: number; unexpected: number; flaky: number; skipped: number };
};

export function collectFailures(report: JsonReport): FailedTest[] {
  const failures: FailedTest[] = [];
  const pathsNamed = (result: JsonResult, name: string) =>
    (result.attachments ?? [])
      .filter((item) => item.name === name && item.path)
      .map((item) => item.path as string);

  const visit = (suite: JsonSuite) => {
    suite.suites?.forEach(visit);
    for (const spec of suite.specs ?? []) {
      for (const test of spec.tests) {
        if (test.status !== 'unexpected') continue;
        const result = test.results[test.results.length - 1];
        failures.push({
          title: spec.title,
          error: stripVTControlCharacters(
            result.errors?.map((item) => item.message ?? '').join('\n') ?? '',
          ),
          tracePaths: pathsNamed(result, 'trace'),
          screenshotPaths: pathsNamed(result, 'screenshot'),
        });
      }
    }
  };
  report.suites.forEach(visit);
  return failures;
}

export class TestRunnerTool {
  constructor(
    private readonly cwd: string,
    private readonly allowExecution: boolean,
  ) {}

  private buildEnv(): NodeJS.ProcessEnv {
    const env: NodeJS.ProcessEnv = {};
    for (const key of SAFE_ENV_KEYS) {
      if (process.env[key] !== undefined) env[key] = process.env[key];
    }
    // An empty value stops dotenv loading the real key from .env.
    env.LLM_API_KEY = '';
    return env;
  }

  run(name: CommandName): Promise<CommandResult> {
    if (!this.allowExecution) {
      return Promise.reject(new GuardrailError('ALLOW_TEST_EXECUTION is not true'));
    }
    return new Promise((done) => {
      execFile(
        process.execPath,
        [...COMMANDS[name]],
        {
          cwd: this.cwd,
          env: this.buildEnv(),
          timeout: LIMITS.commandTimeoutMs,
          maxBuffer: 10_000_000,
        },
        (error, stdout, stderr) => {
          const exitCode = error ? (typeof error.code === 'number' ? error.code : 1) : 0;
          done({ exitCode, output: `${stdout}\n${stderr}` });
        },
      );
    });
  }

  // Keeps the original failure evidence before a rerun replaces it.
  async preserveArtifacts(runId: string): Promise<void> {
    await cp(
      resolve(this.cwd, 'artifacts/test-results'),
      resolve(this.cwd, 'artifacts/runs', runId, 'first-run'),
      { recursive: true },
    );
  }

  async runGeneratedTests(runId: string): Promise<TestRunResult> {
    const startedAt = new Date().toISOString();
    const { exitCode } = await this.run('generatedTests');
    const text = await readFile(resolve(this.cwd, 'artifacts/results.json'), 'utf8');
    const report = JSON.parse(text) as JsonReport;
    const failures = collectFailures(report);
    return {
      runId,
      startedAt,
      finishedAt: new Date().toISOString(),
      exitCode,
      passed: report.stats.expected,
      failed: report.stats.unexpected + report.stats.flaky,
      skipped: report.stats.skipped,
      reportPath: 'playwright-report/index.html',
      tracePaths: failures.flatMap((item) => item.tracePaths),
      screenshotPaths: failures.flatMap((item) => item.screenshotPaths),
      failures,
    };
  }
}

Protection	How it works
Fixed commands	COMMANDS is a constant in code
No shell	execFile runs one program directly
Time limit	A hung run is stopped
Restricted environment	Only SAFE_ENV_KEYS are passed
Execution switch	ALLOW_TEST_EXECUTION must be true

---

🛠️ Read the Results

Playwright’s JSON reporter writes artifacts/results.json. The collectFailures function walks its nested suites and collects each failure with its evidence. The result matches the TestRunResult contract.

---

📌 Test the Executor

Create tests/unit/test-runner.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { collectFailures, TestRunnerTool, type JsonReport } from '../../src/tools/test-runner.tool.js';

const report: JsonReport = {
  stats: { expected: 1, unexpected: 1, flaky: 0, skipped: 0 },
  suites: [
    {
      suites: [
        {
          specs: [
            { title: 'passes', tests: [{ status: 'expected', results: [{}] }] },
            {
              title: 'fails',
              tests: [
                {
                  status: 'unexpected',
                  results: [
                    {
                      errors: [{ message: 'Timeout waiting for button' }],
                      attachments: [{ name: 'trace', path: 'artifacts/trace.zip' }],
                    },
                  ],
                },
              ],
            },
          ],
        },
      ],
    },
  ],
};

test('collects failures from nested suites', () => {
  assert.deepEqual(collectFailures(report), [
    {
      title: 'fails',
      error: 'Timeout waiting for button',
      tracePaths: ['artifacts/trace.zip'],
      screenshotPaths: [],
    },
  ]);
});

test('refuses to run unless execution is enabled', async () => {
  await assert.rejects(new TestRunnerTool('.', false).run('generatedTests'), /ALLOW_TEST_EXECUTION/);
});

Now set ALLOW_TEST_EXECUTION=true in your .env, because you will run the whole workflow in the next lesson.

---

🔑 Never Run a Model’s Command

Compare these two designs, using what you learned in MCP Tool Design for QA.

Unsafe	Safe
exec(modelOutput)	execFile(node, COMMANDS[name])
Any string can run	Three named commands
Shell features are available	No shell is involved
Full environment passed	Short allowlist

---

🌍 Real-Life Example: The Long Test Name

A tool builds a shell command by joining a test name into a string. A crafted title adds ; curl and an address to the command. With execFile and a fixed list, the same text would be a harmless argument, if it even got that far.

---

🧪 Try It Yourself

1. Create the files and run the unit tests.
2. Set ALLOW_TEST_EXECUTION=false, and confirm that the run is refused.
3. Add a smoke command that runs only the smoke test.
4. Run it from a short script.
5. Print the keys that buildEnv allows, and check that no secret is present.

---

📋 Ready-to-Copy Prompt

You are a security reviewer checking a test runner in a QA agent.

Code:
[paste your executor]

Check that model output can never choose the command or its arguments,
that no shell is used, that a time limit applies and that secrets do not
reach the test process.

Return a table with: Risk, Evidence in the Code and Smallest Fix.

---

⚠️ Common Mistakes

• Using exec with a joined string
• Passing the full environment to tests
• Running with no time limit
• Parsing only the console output
• Letting a model pick the command

---

🎯 Key Takeaways

• Execution is fixed, narrow and predictable.
• No shell means no shell injection.
• Pass only the settings that tests need.
• Structured reports beat parsing console text.

---

🎤 Interview Questions

1. How do you safely execute tests from an AI agent?

Model answer: I use a fixed allowlist of commands, run them with no shell and apply a timeout. I restrict the environment and require human approval first.

2. What is the difference between exec and execFile?

Model answer: exec starts a shell, so special characters can run extra commands. execFile starts one program with separate arguments, which avoids that risk.

3. Why restrict environment variables for test runs?

Model answer: Test code should not be able to read secrets that it does not need. Limiting the environment reduces what any bad code could expose.

4. Why use the Playwright JSON reporter?

Model answer: It provides structured results, including errors and attachment paths. That is more reliable than parsing console output.

5. Why separate execution from model reasoning?

Model answer: It keeps results predictable and auditable. A model can suggest what happened, but only fixed code decides what runs.

---

📝 Assignment

Add a maxOutputSize check, so that very long error messages are shortened before they reach the analyst. Write a unit test that proves the limit works. Explain what evidence could be lost and how the trace still preserves it.

---

❓ FAQ

Why call node directly instead of npx?

On Windows, npx is a script that needs a shell. Calling Node with the tool’s path avoids a shell on every system.

Can tests still read secrets from .env themselves?

The validator blocks other process.env access and file imports in generated tests. That is why the layers work together.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Orchestrator: Run the Whole Workflow with a Mock LLM.