Validate AI-Generated Code: Static Checks and Human Approval

This lesson contains the most important safety gate in the project. Before generated code is saved or run, your system must check it automatically, and then a person must approve it. You will validate AI-generated code using plain rules that a model cannot talk its way past.

---

💡 How to Validate AI-Generated Code

Check	What it catches
Output path	Writing outside the approved folder
Syntax	Code that cannot compile
Imports	Access to files, network or shell
Banned patterns	Fixed sleeps, CSS locators, other env variables
Absolute URLs	Tests that leave the approved host
Locator evidence	Elements that nobody observed

---

🔍 Build the Validator

Create src/validators/generated-code.validator.ts.

import ts from 'typescript';
import { isSafeGeneratedPath } from '../core/guardrails.js';
import type { PageObservation } from '../core/types.js';
import type { GeneratedFile } from '../schemas/generation.schema.js';

const BANNED: Array<[RegExp, string]> = [
  [/waitForTimeout/, 'Fixed sleeps are not allowed'],
  [/page\.locator\(/, 'Use role or label locators, not page.locator'],
  [/\b(eval|exec|spawn)\(/, 'Running code or commands is not allowed'],
  [/process\.env\.(?!TEST_USER_EMAIL|TEST_USER_PASSWORD)/, 'Only test credentials may be read'],
  [/https?:\/\//, 'Use relative URLs so tests stay on the approved host'],
];

export function validateGeneratedFile(file: GeneratedFile, observation: PageObservation): string[] {
  const errors: string[] = [];

  if (!isSafeGeneratedPath(file.path)) {
    errors.push(`Unsafe output path: ${file.path}`);
  }

  const { diagnostics = [] } = ts.transpileModule(file.content, { reportDiagnostics: true });
  for (const item of diagnostics) {
    errors.push(`Syntax error: ${ts.flattenDiagnosticMessageText(item.messageText, ' ')}`);
  }

  for (const { fileName } of ts.preProcessFile(file.content).importedFiles) {
    if (fileName !== '@playwright/test') {
      errors.push(`Import is not allowed: ${fileName}`);
    }
  }

  for (const [pattern, message] of BANNED) {
    if (pattern.test(file.content)) {
      errors.push(message);
    }
  }

  const seen = observation.elements;
  for (const [, role, name] of file.content.matchAll(/getByRole\('(\w+)',\s*\{\s*name:\s*'([^']+)'/g)) {
    // A heading can appear only after an action, so headings are checked at run time.
    if (role !== 'heading' && !seen.some((item) => item.role === role && item.name === name)) {
      errors.push(`No evidence for ${role} "${name}"`);
    }
  }
  for (const [, name] of file.content.matchAll(/getByLabel\('([^']+)'\)/g)) {
    if (!seen.some((item) => item.name === name)) {
      errors.push(`No evidence for label "${name}"`);
    }
  }

  return errors;
}

The check is deliberately strict. A false alarm costs a moment of review, while a missed problem could run unsafe code. It builds on your AI agent guardrails.

---

🧩 Test the Validator

Create tests/unit/generated-code.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import type { PageObservation } from '../../src/core/types.js';
import { validateGeneratedFile } from '../../src/validators/generated-code.validator.js';

const observation: PageObservation = {
  url: 'http://127.0.0.1:4321/login',
  title: 'Demo Login',
  objective: 'x',
  elements: [{ role: 'button', name: 'Sign in' }, { role: 'textbox', name: 'Email' }],
};
const path = 'tests/generated/login.spec.ts';
const header = "import { test, expect } from '@playwright/test';\n";
const errorsFor = (content: string, filePath = path) =>
  validateGeneratedFile({ path: filePath, content }, observation);

test('accepts code that follows the rules', () => {
  const body = "test('x', async ({ page }) => { await page.getByRole('button', { name: 'Sign in' }).click(); });";
  assert.deepEqual(errorsFor(header + body), []);
});

test('rejects each kind of unsafe code', () => {
  assert.match(errorsFor(header, '../../.env')[0], /Unsafe output path/);
  assert.match(errorsFor(header + 'const = ;')[0], /Syntax error/);
  assert.match(errorsFor("import fs from 'node:fs';")[0], /Import is not allowed/);
  assert.match(errorsFor(header + 'page.waitForTimeout(5000);')[0], /Fixed sleeps/);
  assert.match(errorsFor(header + 'process.env.LLM_API_KEY;')[0], /test credentials/);
  assert.match(errorsFor(header + "page.goto('https://example.com');")[0], /relative URLs/);
});

test('rejects a locator with no evidence', () => {
  const body = "page.getByRole('button', { name: 'Log in' });";
  assert.deepEqual(errorsFor(header + body), ['No evidence for button "Log in"']);
});

---

🛠️ Add Human Approval

Create src/core/approval.ts.

import { createInterface, type Interface } from 'node:readline/promises';

export type ApprovalAction =
  | 'write-generated-code'
  | 'run-generated-tests'
  | 'apply-healing-patch';

export type ApprovalRequest = { action: ApprovalAction; summary: string };

export type ApprovalRecord = ApprovalRequest & { approved: boolean; decidedAt: string };

export interface Approver {
  decide(request: ApprovalRequest): Promise<boolean>;
  close(): void;
}

export class ConsoleApprover implements Approver {
  private readonly prompt: Interface = createInterface({
    input: process.stdin,
    output: process.stdout,
  });

  async decide({ action, summary }: ApprovalRequest): Promise<boolean> {
    console.log(`\nApproval needed: ${action}\n${summary}`);
    const answer = await this.prompt.question('Type yes to approve: ');
    return answer.trim().toLowerCase() === 'yes';
  }

  close(): void {
    this.prompt.close();
  }
}

export class ScriptedApprover implements Approver {
  readonly requests: ApprovalRequest[] = [];

  constructor(private readonly answers: Partial<Record<ApprovalAction, boolean>>) {}

  async decide(request: ApprovalRequest): Promise<boolean> {
    this.requests.push(request);
    return this.answers[request.action] ?? false;
  }

  close(): void {}
}

Design choice	Reason
Only yes counts	An accidental Enter key does not approve
ScriptedApprover defaults to false	A forgotten test answer can never approve
Each decision is recorded	You keep an audit trail
The full code is shown	A person can really review it

You saw how reviewers can miss things in Testing AI Agents. Keep the approval text short and specific so that people actually read it.

---

🌍 Real-Life Example: The Helpful Import

A generated test imports node:fs to read a settings file. It looks harmless, and it even works. But test code that can read files can also read .env. The import check rejects it, and the model must use the approved environment variables.

---

🧪 Try It Yourself

1. Create the files and run the unit tests.
2. Add a banned pattern for test.only.
3. Write a unit test that proves it works.
4. Pretend to be a reviewer, and write down what you check first.
5. Run npm run typecheck.

---

📋 Ready-to-Copy Prompt

You are a security-focused reviewer of AI-generated test code.

Validator rules:
[paste your validator]

Suggest generated test code that would pass these rules but still be
unsafe or useless. For each example, explain the risk and propose one new
rule or review step.

Only use examples that are safe to run in a test environment.

---

⚠️ Common Mistakes

• Running generated code to see whether it works
• Using a model as the only validator
• Letting an empty answer count as approval
• Showing reviewers only a summary
• Treating text rules as a full sandbox

---

🎯 Key Takeaways

• Check generated code before it is saved or run.
• Validators use plain rules that a model cannot persuade.
• Locators need evidence from the real page.
• A person gives the final approval.

---

🎤 Interview Questions

1. How would you validate AI-generated test code?

Model answer: I would check the path, syntax, imports, banned patterns and locator evidence. Then I would run lint and type checks, and require human approval before execution.

2. Why not ask another model to validate it?

Model answer: A model can be useful for review, but it is not deterministic and can be manipulated. Safety checks should be rules in code, with any model review as an extra layer.

3. How do you design a human approval step?

Model answer: I show the exact action and its content, require an explicit positive answer and record the decision. The system must default to refusing.

4. Are static checks enough to make generated code safe?

Model answer: No. They reduce risk but can be bypassed. I combine them with limited permissions, test-only environments, human review and restricted execution.

5. How do you test a validator?

Model answer: I feed it examples of each kind of bad code and confirm that each is rejected. I also check that a valid file passes, to catch overly strict rules.

---

📝 Assignment

Add a rule that requires every generated test title to begin with a scenario ID such as TS-1. Write one passing and one failing unit test. Then update the generator’s conventions to match.

---

❓ FAQ

Is one approval enough?

This project asks twice: before writing and before running. Each is a different risk, so each gets its own decision.

Will a strict validator reject good code?

Sometimes. Review those cases, and adjust the rule only when you understand the risk.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Test Executor: Run Playwright Tests in a Controlled Way.