Review AI-Generated Automation Tests: Catch Weak Playwright Code

AI-generated test code usually looks tidy, which is exactly why it needs a careful reader. In this final lesson of Level 4 you will learn how to review AI-generated automation tests, spot common weak patterns and prove that each test protects something real. You will finish with a practice review and answer key.

---

💡 How to Review AI-Generated Automation Tests

---

🔍 The Eight-Point Review Checklist

Check	What to ask
Purpose	Does the title match what the test does?
Requirement	Does the expected result come from a real source?
Locators	Are they stable, and do they exist?
Assertions	Do they prove the outcome?
Waits	Are fixed waits avoided?
Independence	Can it run alone and in any order?
Data and secrets	Is the data safe and free of passwords?
Failure	Have you seen it fail for the right reason?

---

🧩 Weak Patterns to Spot

• Assertions that always pass
• Only checking that something is visible
• try and catch blocks that swallow failures
• Missing await on actions or assertions
• test.only left in the code
• Hard-coded credentials
• A changed expected result with no explanation

---

🛠️ Practice: Find the Planted Mistakes

Read this test carefully. It contains several problems.

import { test, expect } from '@playwright/test';

test.only('user can save a profile', async ({ page }) => {
  await page.goto('https://example.com/profile');
  await page.locator('div:nth-child(3) > input').fill('Asha');
  await page.getByLabel('Password').fill('RealPassword123');
  page.getByRole('button', { name: 'Save' }).click();
  await page.waitForTimeout(5000);

  try {
    await expect(page.getByRole('button', { name: 'Save' })).toBeVisible();
  } catch (error) {
    console.log('Save button not found');
  }
});

Answer Key

Problem	Why it matters	Fix
test.only	Skips other tests in the file	Remove it
Position-based CSS	Breaks when the layout changes	Use a label or role
Hard-coded password	Exposes a secret	Use a test secret from configuration
Missing await on click	The action may not finish in order	Add await
waitForTimeout(5000)	Slow and unreliable	Assert the expected state
Checks only the Save button	Does not prove anything was saved	Assert the saved result
try and catch	The test can never fail here	Remove it

---

📌 Prove It Can Fail

For each important test, try a controlled change, such as a wrong expected value or a mocked error response. If the test still passes, it needs work.

Our Network Mocking lesson shows how to simulate failed responses safely. For timing problems, see Flaky Test Diagnosis with AI.

---

🔑 Enforce What You Can Automatically

Humans should focus on judgment. Tools can catch routine problems, such as linting rules for missing await or a CI check that fails on test.only. Playwright has a forbidOnly setting:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  forbidOnly: !!process.env.CI,
});

---

💡 Record Your Review

Keep a short note in your pull request with what AI generated, what you changed and how you proved the test works. Reviewers then know what to focus on.

---

🧪 Try It Yourself

1. Take a test you generated earlier in this level.
2. Score it against the eight-point checklist.
3. Run the prompt below in a new chat for a second opinion.
4. Fix the important issues.
5. Prove that the fixed test can fail.

---

📋 Ready-to-Copy Prompt

Act as a strict senior test automation reviewer. Review this Playwright
TypeScript test. Do not rewrite it.

Requirement:
[paste requirement]

Test:
[paste test]

Check purpose, requirement match, locators, assertions, waits,
independence, test data, secrets and missing await.

Return a table with: Line, Issue, Severity, Why It Matters, Suggested Fix.

Then answer: could this test pass while the feature is broken? Explain
how to prove it can fail.

---

⚠️ Common Mistakes

• Approving a test because it is green
• Judging the code only by style
• Allowing failures to be caught and ignored
• Accepting changed expected results without evidence
• Reviewing only the new lines and not their effect

---

🎯 Key Takeaways

• Clean-looking code can still test very little.
• Use the same checklist on every generated test.
• See each test fail for the right reason.
• Automate routine checks, and keep human judgment for meaning.

---

🎤 Interview Questions

1. How do you review AI-generated test automation code?

Model answer: I check that it matches the requirement, then review locators, assertions, waits, independence, data and secrets. I run it and confirm it fails when the behaviour is broken.

2. What is a false positive and a false negative in test automation?

Model answer: Terminology varies, but commonly a false positive reports a failure when the product is fine, and a false negative passes when a defect exists. False negatives are dangerous because they create false confidence.

3. What is mutation testing?

Model answer: Mutation testing makes small deliberate changes to code and checks whether tests detect them. If tests still pass, they may not be protecting that behaviour.

4. Why is try and catch dangerous in a test?

Model answer: It can swallow an assertion failure, which allows the test to pass when it should fail. Error handling in tests must not hide failures that the test exists to detect.

5. What does forbidOnly do in Playwright?

Model answer: It causes the run to fail if test.only is found in the code. Teams often enable it in CI, so that a focused test cannot accidentally skip the rest of the suite.

---

📝 Assignment

Review five AI-generated tests with the checklist, and log each finding. Prove that at least three can fail. This completes Level 4, and you are ready to move on to testing AI products in Level 5.

---

❓ FAQ

Should AI review its own code?

It can give a useful second opinion, ideally in a separate session. A person must still make the final decision.

How much review is enough?

Match the review to the risk. A test covering payments needs more care than a simple display check, and every test should be seen failing once.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with How to Test AI Applications: Why AI Testing Is Different.