Testing RAG Applications: Retrieval, Grounding and Answers

Many company chatbots answer from their own documents. That design is called RAG. When its answer is wrong, the fault could be in the documents, the search or the model. In this lesson you will learn testing RAG applications one stage at a time, so you can find where the problem really is.

📖 What You’ll Learn

  • What RAG is and how it works
  • How to test retrieval separately from the answer
  • How to calculate precision and recall
  • How to diagnose where a failure happened

💡 What Is RAG?

Explain Like I’m 5: Open-Book Test

Imagine: In an open-book test, you find the right page first, and then write your answer.

What it really means: RAG finds relevant documents, then the model answers from them.

In testing: The chatbot looks up your returns policy before replying.

RAG stands for Retrieval-Augmented Generation. It has three main stages:

StageWhat happensWhat can go wrong
DocumentsContent is prepared and split into chunksOld, missing or badly split content
RetrievalThe system finds matching chunksFinds the wrong chunks, or misses the right one
GenerationThe model writes an answer from themIgnores the source or adds to it

🔍 Testing RAG Applications One Stage at a Time

Explain Like I’m 5: Finding the Broken Step

Imagine: If your cake is bad, you check the recipe, the shopping and the baking one at a time.

What it really means: Test each stage so that you can find the real fault.

In testing: A wrong answer may be a search problem, not a model problem.


🧩 Test the Documents

Check that the source content is correct, current and approved. Check that private documents are not indexed for the wrong users. A perfect system still gives a wrong answer if its source is out of date.


🛠️ Test Retrieval

For each test question, a person lists which documents should be found. You can then measure the search on its own.

Explain Like I’m 5: Picking Apples

Imagine: You pick three fruits from a basket. Precision asks how many were apples. Recall asks how many of all the apples you found.

What it really means: Precision checks for junk, and recall checks for missing items.

In testing: Good retrieval finds the right chunks and little else.

retrieved = ["doc3", "doc7", "doc1", "doc9", "doc2"]
relevant = {"doc1", "doc3", "doc5"}
k = 3

top_k = retrieved[:k]
hits = [doc for doc in top_k if doc in relevant]

precision = len(hits) / k
recall = len(hits) / len(relevant)

print(f"Hits in top {k}: {hits}")
print(f"Precision@{k}: {precision:.2f}")
print(f"Recall@{k}: {recall:.2f}")
Hits in top 3: ['doc3', 'doc1']
Precision@3: 0.67
Recall@3: 0.67

Two of the top three results are useful, and two of the three relevant documents were found. doc5 was missed, so an answer that needs it may be incomplete.


📌 Test the Generated Answer

Common checks include:

CheckQuestion
FaithfulnessIs every claim supported by the retrieved text?
RelevanceDoes it answer the question?
Unknown handlingDoes it admit when the documents lack the answer?
CitationsDo the links really support the claim?
ConflictsWhat happens when two documents disagree?

Learn the scoring ideas in LLM Evaluation Metrics for Testers.


🔑 Diagnose the Failure

Right document found?Answer correct?Likely problem
NoNoRetrieval or document content
YesNoGeneration or prompt
NoYesLucky, or model used outside knowledge
YesYesWorking as intended

The third row deserves attention. A correct answer without the right source may mean that the model used its general knowledge, which it should not do for company policy.

Test access rules

Retrieval can leak data. Confirm that a normal user can never receive content from documents meant only for staff or other customers.


🌍 Real-Life Example: The Old Policy

A chatbot says returns are accepted for 30 days, but the current policy is 14 days. The tester checks what was retrieved, and finds the old PDF ranked first. The model was faithful to its source. The real fix is to remove the old document, and add a test for that question.


🧪 Try It Yourself

  1. Write three short made-up policy documents.
  2. Write five questions, and list which document answers each.
  3. Pretend a search returned certain documents, and calculate precision and recall.
  4. Use the prompt below to create test questions.
  5. Add one question that the documents cannot answer.

📋 Ready-to-Copy Prompt

You are a QA engineer testing a RAG application.

Approved documents:
[paste labelled documents with no private data]

Create a test table with: Question, Expected Document IDs, Required
Facts, Must Not Claim and Test Type.

Include direct questions, reworded questions, questions needing two
documents, questions with conflicting sources, and questions the
documents cannot answer.

Rules:
- Use only the documents above.
- For an unanswerable question, set the expected behaviour to
  "says the information is not available".

⚠️ Common Mistakes

  • Evaluating only the final answer
  • Forgetting to test outdated documents
  • Assuming a cited source supports the claim
  • Skipping questions that have no answer
  • Forgetting document access rules

🎯 Key Takeaways

  • RAG has document, retrieval and generation stages.
  • Test each stage to find the true fault.
  • Precision checks for junk, and recall checks for gaps.
  • Retrieval must respect user access rules.

🎤 Interview Questions

1. What is RAG?

Model answer: RAG stands for Retrieval-Augmented Generation. The system finds relevant content from a knowledge base and gives it to the model, so that the answer can be based on that information.

2. How would you test a RAG application?

Model answer: I would test the documents, retrieval and generation separately. I measure whether the expected chunks are found, whether answers are supported and relevant, and how the system handles unknown or conflicting information.

3. What is the difference between precision and recall in retrieval?

Model answer: Precision is the share of retrieved items that are relevant. Recall is the share of all relevant items that were retrieved. Together, they show junk and missing content.

4. If a RAG answer is wrong, how do you find the cause?

Model answer: I inspect the retrieved chunks first. If the right content was missing, I investigate documents or retrieval. If it was present and the answer was still wrong, I examine the prompt and generation.

5. What security risks are specific to RAG?

Model answer: Retrieval may expose documents that a user should not access, and retrieved content can contain malicious instructions. I test access controls and treat document text as untrusted.


📝 Assignment

Build a small RAG test plan with ten questions and expected documents. Calculate precision and recall for at least three made-up searches. Include two access-control tests and one prompt injection test inside a document.


❓ FAQ

What is a chunk?

A chunk is a small section of a document that is stored and searched. Poor chunking can split important information apart.

Do I need access to the vector database?

It helps to see what was retrieved. Ask developers to log retrieved document IDs in your test environment.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Testing AI Agents: Tool Calls, Loops and Stop Conditions.

LLM Evaluation Metrics for Testers: Faithfulness, Relevance and Hallucination Rate

“The chatbot seems good” is not a measurement. Numbers let you compare versions and spot problems, but only if you understand what they mean. In this lesson you will learn the main LLM evaluation metrics for testers, calculate them simply and avoid being misled by one neat score.

📖 What You’ll Learn

  • What faithfulness, relevance and hallucination rate mean
  • How to calculate simple scores
  • How AI judges work and where they fail
  • How to report results honestly

💡 The Main LLM Evaluation Metrics

Explain Like I’m 5: Marking Homework

Imagine: One mark is for answering the question asked. Another mark is for using facts from the book.

What it really means: Different metrics measure different kinds of quality.

In testing: An answer can be polite and still be wrong.

MetricQuestion it answersExample failure
RelevanceDid it answer the question asked?Talks about delivery when asked about returns
FaithfulnessIs it supported by the source?Adds a rule that is not in the policy
CorrectnessDoes it match the expected answer?Says 60 days instead of 30
Hallucination rateHow often does it make things up?Invents an order number
Refusal accuracyDoes it refuse the right things?Refuses a normal question
Latency and costIs it fast and affordable?Too slow or expensive at scale

🔍 Faithful Is Not the Same as Correct

Explain Like I’m 5: Copying the Wrong Book

Imagine: You copy perfectly from a book, but the book is out of date. You were faithful, and still wrong.

What it really means: An answer can match its source when the source itself is wrong.

In testing: Check the documents as well as the answer.


🧩 A Simple Calculation

Here, a reviewer has labelled five answers. This is made-up data.

reviews = [
    {"relevant": True, "faithful": True},
    {"relevant": True, "faithful": False},
    {"relevant": False, "faithful": True},
    {"relevant": True, "faithful": True},
    {"relevant": True, "faithful": True},
]

total = len(reviews)
relevance = sum(r["relevant"] for r in reviews) / total
faithfulness = sum(r["faithful"] for r in reviews) / total
hallucination_rate = 1 - faithfulness

print(f"Relevance: {relevance:.0%}")
print(f"Faithfulness: {faithfulness:.0%}")
print(f"Hallucination rate: {hallucination_rate:.0%}")
Relevance: 80%
Faithfulness: 80%
Hallucination rate: 20%

Five answers are too few to trust, so this only demonstrates the arithmetic. Define your own rules for what counts as a hallucination, and use them consistently.


🛠️ AI Judges

Explain Like I’m 5: A Student Marking a Test

Imagine: A clever student can help mark papers, but the teacher checks some to be sure.

What it really means: An AI judge scores answers, and people must check the judge.

In testing: Compare its scores with human labels on a sample.

An AI judge is a model that scores another model’s answers against written criteria. It is fast and handles meaning, but it can be wrong, inconsistent or too generous.

To check a judge:

  1. Label a sample of answers yourself.
  2. Run the judge on the same sample.
  3. Count where you disagree.
  4. Improve the rubric, or add human review.
  5. Check it again when the judge model changes.

📌 Do Not Trust One Number

A single score can hide serious problems.

CategoryPass rateCases
Delivery98%100
Returns94%100
Privacy60%5

These numbers are made up. The overall pass rate is about 95%, which looks strong. Privacy has only 5 cases, so its low score barely moves the average. Always report by category and by risk, and show the number of cases.

Small samples mislead

Moving from 18 out of 20 to 19 out of 20 may be simple chance. Repeat runs and compare like with like before claiming an improvement.


🌍 Real-Life Example: A Fake Improvement

A team changes a prompt, and the average score rises by two points. Looking closer, the model has started refusing difficult questions. Refused answers avoid hallucinations, so faithfulness goes up. But users now get less help. Tracking refusal accuracy reveals the trade-off.


🧪 Try It Yourself

  1. Use three answers from a chatbot, or write made-up ones.
  2. Label each for relevance and faithfulness by hand.
  3. Run the prompt below as an AI judge.
  4. Note every place where you disagree.
  5. Improve the rubric, and try again.

📋 Ready-to-Copy Prompt

You are evaluating a chatbot answer. Use only the source and rubric below.

Question: [question]
Source text: [approved source]
Answer: [answer to evaluate]

Rubric:
- Relevant: the answer addresses the question asked.
- Faithful: every factual claim is supported by the source.

Return JSON with these fields: relevant (true or false), faithful
(true or false), unsupported_claims (list), reason (one sentence).

If the source does not contain enough information, say so in the reason.
Do not use outside knowledge.

⚠️ Common Mistakes

  • Reporting one average score
  • Assuming a higher score always means a better product
  • Trusting an AI judge without checking it
  • Comparing runs that used different datasets
  • Ignoring the number of test cases

🎯 Key Takeaways

  • Use several metrics, because each measures something different.
  • Faithful answers can still be wrong if the source is wrong.
  • Check AI judges against human labels.
  • Report by category, and show your sample size.

🎤 Interview Questions

1. What is the difference between faithfulness and relevance?

Model answer: Faithfulness checks whether claims are supported by the provided source. Relevance checks whether the response addresses the user’s question. An answer can be faithful but irrelevant, or relevant but unsupported.

2. How would you measure hallucination rate?

Model answer: I would define what counts as unsupported, then evaluate a representative set of answers using humans or a validated judge. The rate is the share containing unsupported claims, reported by category.

3. What is an LLM judge, and what are its limits?

Model answer: It is a model that scores outputs against criteria. It scales well, but it can be inconsistent or biased. I compare it with human labels and review risky or uncertain results.

4. Why can one evaluation score be misleading?

Model answer: An average can hide failures in small but critical categories. It can also improve for the wrong reason, such as refusing more questions. I report by category, risk and sample size.

5. How do you compare two prompt versions fairly?

Model answer: I use the same dataset, settings and judge, and repeat runs where variation matters. I compare by category and check trade-offs such as accuracy, refusals, latency and cost.


📝 Assignment

Evaluate 15 answers by hand, then with an AI judge. Calculate how often you agreed and report results by category. Write three sentences explaining what the numbers do and do not prove. Review AI hallucinations to decide what counts as one, and use your golden dataset as input.


❓ FAQ

Which metric is most important?

It depends on the product and its risks. For a policy chatbot, correctness and faithfulness are often top priorities.

Do I need a paid tool to calculate these?

No. You can start with human labels and a spreadsheet, and add tools as your needs grow.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Testing RAG Applications: Retrieval, Grounding and Answers.

Golden Dataset for LLM Evaluation: Build Your Test Set

You cannot measure an AI feature without good examples. A golden dataset is a trusted set of test cases that you run again and again. In this lesson you will build a golden dataset for LLM evaluation, cover the cases that matter and keep it trustworthy over time.

📖 What You’ll Learn

  • What a golden dataset is
  • What each test case should contain
  • How to cover important and risky situations
  • How to maintain the set over time

💡 What Is a Golden Dataset?

Explain Like I’m 5: Answer Sheet

Imagine: A teacher keeps one checked answer sheet, and marks every student’s paper against it.

What it really means: A golden dataset is a trusted set of inputs and expected outcomes.

In testing: You rerun the same examples after every change.

An AI answer can vary, so “expected” often means required facts and rules. It does not have to mean exact words.


🔍 What Goes in Each Case

FieldPurposeExample
IDTracks the caseR1
InputWhat the user saysHow many days can I return a shirt?
Must includeFacts required30 days
Must not includeForbidden claimsany time
CategoryGroups resultsreturns
RiskSets strictnesshigh
SourceProves the answerReturns policy, section 2

🧩 Golden Dataset Coverage

Explain Like I’m 5: A Balanced Lunchbox

Imagine: A lunchbox with only biscuits is not a good lunch. You want a bit of everything.

What it really means: A good set covers different kinds of questions.

In testing: Include easy, hard, unusual and unsafe cases.

Case typeExample
Common questionsThe questions users ask most
Hard wordingTyping errors and vague phrasing
Edge cases“I bought it exactly 30 days ago”
Unknown answersQuestions the sources cannot answer
Out of scope“What is the weather today?”
SafetyRequests for private data
Real failuresBad answers from production

🛠️ Check the Dataset Itself

A dataset can have bugs too. This Python script checks that every case has its required fields, and shows the balance of categories and risks. It uses made-up data.

from collections import Counter

golden_set = [
    {"id": "R1", "question": "How many days can I return a shirt?",
     "must_include": ["30 days"], "category": "returns", "risk": "high"},
    {"id": "R2", "question": "Can I return a used item?",
     "must_include": ["unused"], "category": "returns", "risk": "high"},
    {"id": "D1", "question": "Do you deliver on Sundays?",
     "must_include": ["no"], "category": "delivery", "risk": "medium"},
    {"id": "O1", "question": "What is the weather today?",
     "must_include": ["cannot help"], "category": "out_of_scope", "risk": "low"},
]

required = {"id", "question", "must_include", "category", "risk"}
for case in golden_set:
    missing = required - case.keys()
    assert not missing, f"{case['id']} is missing {sorted(missing)}"

print(Counter(case["category"] for case in golden_set))
print(Counter(case["risk"] for case in golden_set))
Counter({'returns': 2, 'delivery': 1, 'out_of_scope': 1})
Counter({'high': 2, 'medium': 1, 'low': 1})

This simple check shows immediately if a category has too few examples.


📌 How AI Can Help

Explain Like I’m 5: Extra Practice Questions

Imagine: A friend can make up practice questions, but your teacher still checks them.

What it really means: AI can suggest more cases, and experts must approve them.

In testing: Generate variations, then verify every expected answer.

AI is useful for creating different wordings of the same question. But if a model writes both the questions and the answers, it may repeat its own mistakes. Base expected answers on approved sources, and have a subject expert review them. The checks in How to Review AI-Generated Test Cases work well here too.

Protect people’s data

Real conversations are valuable but can contain personal information. Follow your company’s rules, remove personal details and limit access.


🔑 Keep It Fresh

  • Add every important production failure as a new case.
  • Update expected answers when a policy changes.
  • Remove cases that no longer apply.
  • Record who approved each change, and when.
  • Keep the file under version control.

🌍 Real-Life Example: A Stale Answer

A company changes its returns period from 30 days to 14 days. The bot is updated correctly, but the golden dataset still expects 30. Suddenly, correct answers are marked as failures. A tester checks the source and updates the dataset. Without a named owner, the team might have rolled back a correct change.


🧪 Try It Yourself

  1. Write a small policy of five rules for a made-up shop.
  2. Run the prompt below to draft 12 cases.
  3. Check every case against your policy.
  4. Add one case that the AI missed.
  5. Run the Python check above using your cases.

📋 Ready-to-Copy Prompt

You are helping me build a golden dataset for evaluating a chatbot.

Approved source text:
[paste approved text with no personal data]

Create [number] test cases as a table with: ID, User Input, Must Include,
Must Not Include, Category, Risk and Source Quote.

Include common, vague, edge-case, unknown-answer, out-of-scope and safety
cases.

Rules:
- Base every expected answer on the source text.
- If the source cannot answer, set Must Include to "says it does not know".
- Quote the source line for each case.
- Do not invent policy details.

⚠️ Common Mistakes

  • Including only easy, happy-path questions
  • Letting AI write unreviewed expected answers
  • Forgetting to update the set when policies change
  • Using private customer conversations carelessly
  • Measuring only an overall score without categories

🎯 Key Takeaways

  • A golden dataset is a trusted, repeatable test set.
  • Expected results are often facts and rules, not exact words.
  • Cover common, edge, unknown and unsafe cases.
  • Review and update the dataset like any test asset.

🎤 Interview Questions

1. What is a golden dataset?

Model answer: It is a curated set of inputs with trusted expected outcomes used to evaluate an AI system repeatedly. It lets a team compare versions and detect regressions.

2. What makes a good LLM evaluation dataset?

Model answer: It represents real use and covers common, difficult, unknown, out-of-scope and safety cases. Expected outcomes are reviewed by experts, linked to sources and kept up to date.

3. Can you use AI to generate an evaluation dataset?

Model answer: Yes, especially for variations in wording. I base it on approved sources and have people review expected outcomes, because a model can introduce errors or shared blind spots.

4. How do you maintain a golden dataset?

Model answer: I version it, assign an owner and add real production failures. I update it when policies change and remove obsolete cases, while recording why each change was made.

5. Why should you report evaluation results by category?

Model answer: An overall score can hide a serious weakness. A chatbot may score well overall but fail safety or billing questions, so category results show where the real risk lies.


📝 Assignment

Build a golden dataset of 25 cases for a made-up support bot. Include at least five edge cases and three unknown-answer cases. Have someone else review ten expected answers, and record what changed. Keep it in Git, as you learned in our Git and GitHub Tutorial.


❓ FAQ

How big should my dataset be?

Start small with your most important cases, then expand. Quality and coverage matter more than a large number.

Should I use the same data to improve prompts and to evaluate them?

Keep a separate set for final checks. Otherwise, a prompt can become tuned to your test questions and appear better than it really is.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Evaluation Metrics for Testers: Faithfulness, Relevance and Hallucination Rate.

LLM Test Strategy: How to Test a Chatbot or AI Feature

Testing a chatbot by chatting with it for ten minutes is a demo, and it is not a test. You need a plan that says what good looks like and how you will measure it. In this lesson you will build a practical LLM test strategy for a chatbot or any other AI feature.

📖 What You’ll Learn

  • How to define what a chatbot must and must not do
  • How to set risk levels for different answers
  • Which test types a chatbot needs
  • How to decide when it is ready to release

💡 Start with Purpose and Limits

Explain Like I’m 5: A Shop Helper

Imagine: A new shop helper is told, “Answer questions about shoes. Do not give medical advice.”

What it really means: Decide what the AI should do and what it must refuse.

In testing: A returns chatbot should not offer legal or medical advice.

Write these down before testing:

QuestionExample
What is it for?Answer returns and delivery questions
Who uses it?Shoppers, including new customers
What sources can it use?The approved help centre
What must it refuse?Medical, legal and off-topic requests
When should it hand over?Complaints or anything it cannot answer

🔍 LLM Test Strategy: Set Risk Levels

Explain Like I’m 5: Different Rules

Imagine: Spilling juice gets a wipe. Touching a hot stove gets a very firm no.

What it really means: Not all mistakes are equally serious.

In testing: A clumsy sentence is minor, but leaking data is critical.

RiskExample failureSuggested rule
CriticalReveals another customer’s dataZero tolerance
HighStates the wrong refund periodVery high pass rate, with human review
MediumGives an incomplete answerTracked with an agreed target
LowSlightly awkward toneMonitored

Your team must set the actual numbers.


🧩 Test Types a Chatbot Needs

Test typeExample
Normal functionalChat widget opens and sends a message
Knowledge accuracyAnswers match the policy
Out of scopeRefuses unrelated questions politely
Unknown answersSays it does not know instead of inventing
Multi-turnRemembers details from earlier in the chat
SafetyHandles abuse and harmful requests correctly
HandoverPasses the user to a person when needed
Non-functionalResponse time, cost and rate limits

🛠️ Testing Conversations

Explain Like I’m 5: Remembering the Game

Imagine: If you say, “My name is Asha,” and your friend forgets one minute later, the game feels broken.

What it really means: A conversation has memory, so testing one question is not enough.

In testing: Check that the bot uses details given earlier.

Example conversation test:

  1. User: “I bought running shoes last week.”
  2. User: “Can I return them?”
  3. Check: the answer talks about returning the shoes, without asking what was bought.

📌 Mix Three Kinds of Checking

Explain Like I’m 5: Three Helpers

Imagine: A ruler measures height. A friend says if your drawing looks good. A teacher makes the final call.

What it really means: Use rules, AI judges and people together.

In testing: Each one catches what the others miss.

MethodGood forWeakness
Rule-based checksRequired words, format, banned contentCannot judge meaning
AI judgeRelevance, tone, meaningCan be wrong or inconsistent
Human reviewHard cases and final judgmentSlow and costly

🔑 Release Gates

A clear gate avoids last-minute arguments. Here is an example. Your own numbers should come from your risk levels.

  • All critical safety cases pass.
  • High-risk knowledge cases meet the agreed pass rate.
  • Unknown-answer cases do not invent information.
  • Normal functional tests pass.
  • Human reviewers have checked a sample of answers.

Testing continues after release

Real users will ask things you never imagined. Plan to monitor conversations safely, and add real failures to your test set.


🌍 Real-Life Example: A Risky Gap

A delivery chatbot passes all its product questions. Then a user asks, “Where is my mother’s order?” The test plan never covered requests about another person’s account. A strategy that lists privacy risks would have included this case, and tested that the bot asks for proper verification.


🧪 Try It Yourself

  1. Pick a support bot idea, or use the Dummy E-commerce Website as your made-up store.
  2. Write its purpose, sources and limits.
  3. Run the prompt below.
  4. Assign your own risk levels.
  5. Write a release gate that a manager could understand.

📋 Ready-to-Copy Prompt

You are a senior QA lead creating a test strategy for an LLM chatbot.

Purpose: [purpose]
Users: [users]
Approved knowledge sources: [sources]
Must refuse: [topics]
Handover rules: [rules]

Create a test strategy with: Risks by level, Test Types, 3 example test
cases for each type, Checking Method (rule, AI judge or human), Suggested
Release Gates and Production Monitoring ideas.

Rules:
- Use only the details above.
- Do not invent target numbers. Write "[TEAM TO SET]" for each threshold.
- List missing information as questions.

⚠️ Common Mistakes

  • Testing only friendly, expected questions
  • Forgetting multi-turn conversations
  • Using one pass rate for all kinds of failures
  • Trusting an AI judge without human checks
  • Stopping testing at release

🎯 Key Takeaways

  • Define the purpose, sources and limits first.
  • Set stricter rules for higher-risk failures.
  • Combine rule-based checks, AI judges and people.
  • Keep testing after launch with real examples.

🎤 Interview Questions

1. How would you test a customer support chatbot?

Model answer: I would define its purpose, sources and limits, then assess risks. I would test accuracy, unknown answers, out-of-scope requests, conversations, safety, handover and normal application behaviour, with clear gates for release.

2. How do you decide acceptance thresholds for an AI feature?

Model answer: I base them on risk and agree them with stakeholders. Critical safety failures may have zero tolerance, while lower-risk quality issues can have target pass rates that are tracked over time.

3. What is multi-turn testing?

Model answer: It tests a full conversation rather than one message. I check whether the system retains context, handles corrections and stays consistent as the conversation continues.

4. How can you check AI answers at scale?

Model answer: I combine rule-based checks, AI judges and human review. Rules catch known problems, AI judges help with meaning, and people review risky or uncertain results and check the judge.

5. What should happen when a chatbot does not know the answer?

Model answer: It should say so clearly and offer a safe next step, such as a help article or a human agent. I test this with questions that are outside its approved knowledge.


📝 Assignment

Write a two-page test strategy for a made-up chatbot. Include at least four risk levels, eight test types and a release gate. Compare it with your test plan skills from Lesson 13.


❓ FAQ

How many test cases do I need to start?

Start with a small set covering your critical journeys and risks, then grow it using real failures.

Can one tester own all of this?

Usually, it works best as a team effort involving product, developers and domain experts.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Golden Dataset for LLM Evaluation: Build Your Test Set.

How to Test AI Applications: Why AI Testing Is Different

So far, you have used AI to help you test. Now you will test AI itself. This lesson explains how to test AI applications, why an exact expected result often stops working and which skills you already have that still apply. Your testing mindset matters more here than ever.

📖 What You’ll Learn

  • How AI products differ from normal software
  • Why exact-match checks often fail
  • Which layers of an AI product you can test
  • The new kinds of risk that AI adds

💡 How to Test AI Applications: What Changes

Explain Like I’m 5: A New Kind of Test

Imagine: A maths quiz has one right answer. A story-writing test has many good answers, and a few bad ones.

What it really means: AI answers need to be judged against rules, not matched word for word.

In testing: A chatbot can answer correctly in many different ways.

Remember from How LLMs Work that output can vary between runs. A correct answer can use different words every time.

Normal featureAI feature
One exact expected resultA range of acceptable answers
Same input gives same outputSame input can give different output
Pass or fail per runScores across many examples
Bugs live in codeProblems can also come from prompts, data or the model

🔍 Why Exact Matching Breaks

This small Python example compares an exact check with a check for the important fact.

expected = "Your order ships in 3 days."
answers = [
    "Your order ships in 3 days.",
    "Expect your parcel within 3 days.",
]

for answer in answers:
    print(answer == expected, "3 days" in answer)
True True
False True

Both answers are correct for a customer. The exact check fails one of them, while the fact check passes both. Real evaluation goes further, but the idea is the same: define what must be true, and not the exact words.


🧩 Test the Layers

Explain Like I’m 5: Testing a Sandwich

Imagine: If a sandwich tastes bad, you check the bread, the filling and the sauce one by one.

What it really means: An AI product has several layers, and each can be tested.

In testing: Many failures come from around the model, not from the model.

LayerExample checkType of test
User interfaceThe answer displays correctlyNormal UI test
APIErrors and timeouts are handledNormal API test
Prompt templateVariables are filled in correctlyNormal unit test
Retrieved documentsThe right document is foundRetrieval evaluation
Model answerThe answer is correct and safeAI evaluation
GuardrailsBanned content is blockedRule-based and AI checks

Good news: many of those layers can be tested with the skills you already have from Level 4.


🛠️ New Kinds of Risk

Explain Like I’m 5: A Very Confident Friend

Imagine: A friend answers every question boldly, even when guessing. You love them, but you check before believing them.

What it really means: AI can be wrong, biased or tricked while sounding sure.

In testing: A chatbot may invent a refund policy.

  • Hallucination: it states something false
  • Prompt injection: hidden text changes its behaviour
  • Data leakage: it reveals private information
  • Bias: it treats groups of people unfairly
  • Drift: quality changes after a model or data update

You will test each of these later in this level.

Hard rules can still be strict

Variable output does not mean anything goes. “Never share another customer’s data” is a pass or fail rule, and a single failure matters.


🌍 Real-Life Example: A Returns Chatbot

A shop adds a chatbot that answers returns questions. The policy allows 30 days.

AnswerVerdict
“You have 30 days to return it.”Pass
“Returns are accepted within a month of purchase.”Check: is a month the same as 30 days?
“You have 60 days.”Fail: wrong fact
“Sure, return it any time!”Fail: invented policy

The second answer shows why human judgment still matters. A tester has to decide if it is acceptable, then make that rule clear.


🧪 Try It Yourself

  1. Think of a chatbot on a website you use.
  2. Write one question you would ask it.
  3. Run the prompt below in three new chats.
  4. Mark each answer as pass, fail or unclear.
  5. Write the rule that would make your “unclear” answers easy to judge.

📋 Ready-to-Copy Prompt

You are a senior QA engineer learning to test AI features.

AI feature: [what it does]
Who uses it: [users]
Business rules: [rules]

Create a table of 8 acceptance rules for this feature.
For each rule, give: Rule, Must Always Be True or Must Never Happen,
Example Pass, Example Fail, How It Could Be Checked.

Use only the details above. List anything unclear as questions.

⚠️ Common Mistakes

  • Using exact text matches for every AI answer
  • Testing only the model and ignoring the layers around it
  • Saying “AI is unpredictable, so we cannot test it”
  • Running one example and calling the feature good
  • Treating safety rules as if they were flexible

🎯 Key Takeaways

  • AI answers are judged against rules, not exact text.
  • Quality is measured across many examples.
  • Many layers can still be tested in the usual way.
  • Some rules, especially safety rules, stay strict.

🎤 Interview Questions

1. How is testing an AI application different from testing normal software?

Model answer: Output can vary, and several answers may be correct. I define acceptance criteria and evaluate across many examples. I also test the surrounding layers normally, and I add checks for hallucination, safety, data leakage and drift.

2. Why do exact string assertions often fail for LLM features?

Model answer: The model can express the same correct meaning in different words, so exact matches give false failures. I check required facts, structure and rules, and I use semantic or human review where meaning matters.

3. Which parts of an LLM application can you test deterministically?

Model answer: I can test the UI, APIs, error handling, prompt construction, output parsing, access control and some guardrails with normal assertions. Those layers often cause real production failures.

4. What are the main risks in an AI feature?

Model answer: The main risks include hallucination, prompt injection, data leakage, bias, unsafe content and quality drift after updates. The priority depends on the domain and potential harm.

5. How would you define a pass for a chatbot answer?

Model answer: I would list required facts, forbidden content, tone and format, with agreed rules for borderline cases. I would then measure results across a representative set, with strict limits for critical safety rules.


📝 Assignment

Choose an AI feature you know. Divide it into layers, then write one test idea for each. Mark which ones can use exact checks and which need evaluation or human judgment.


❓ FAQ

Do I need to understand machine learning maths?

No. You need clear rules, good examples and careful measurement, which are all testing skills.

Can I test an AI feature I did not build?

Yes. You can test its behaviour from the outside. Access to prompts and logs makes diagnosis easier.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Test Strategy: How to Test a Chatbot or AI Feature.

Review AI-Generated Automation Tests: Catch Weak Playwright Code

AI-generated test code usually looks tidy, which is exactly why it needs a careful reader. In this final lesson of Level 4 you will learn how to review AI-generated automation tests, spot common weak patterns and prove that each test protects something real. You will finish with a practice review and answer key.

📖 What You’ll Learn

  • A practical code review checklist
  • Common weak patterns in generated tests
  • How to prove a test detects a real break
  • How to record review decisions

💡 How to Review AI-Generated Automation Tests

Explain Like I’m 5: A Shiny Apple

Imagine: An apple can look perfect and still have a worm inside. You cut it open to check.

What it really means: Code can look clean and still test nothing useful.

In testing: A green test is only useful if it checks the right thing.


🔍 The Eight-Point Review Checklist

CheckWhat to ask
PurposeDoes the title match what the test does?
RequirementDoes the expected result come from a real source?
LocatorsAre they stable, and do they exist?
AssertionsDo they prove the outcome?
WaitsAre fixed waits avoided?
IndependenceCan it run alone and in any order?
Data and secretsIs the data safe and free of passwords?
FailureHave you seen it fail for the right reason?

🧩 Weak Patterns to Spot

Explain Like I’m 5: The Empty Lunchbox

Imagine: A lunchbox can have your name on it and still have nothing inside.

What it really means: A test can have a good title and check nothing.

In testing: Look for actions with no meaningful assertion.

  • Assertions that always pass
  • Only checking that something is visible
  • try and catch blocks that swallow failures
  • Missing await on actions or assertions
  • test.only left in the code
  • Hard-coded credentials
  • A changed expected result with no explanation

🛠️ Practice: Find the Planted Mistakes

Read this test carefully. It contains several problems.

import { test, expect } from '@playwright/test';

test.only('user can save a profile', async ({ page }) => {
  await page.goto('https://example.com/profile');
  await page.locator('div:nth-child(3) > input').fill('Asha');
  await page.getByLabel('Password').fill('RealPassword123');
  page.getByRole('button', { name: 'Save' }).click();
  await page.waitForTimeout(5000);

  try {
    await expect(page.getByRole('button', { name: 'Save' })).toBeVisible();
  } catch (error) {
    console.log('Save button not found');
  }
});

Answer Key

ProblemWhy it mattersFix
test.onlySkips other tests in the fileRemove it
Position-based CSSBreaks when the layout changesUse a label or role
Hard-coded passwordExposes a secretUse a test secret from configuration
Missing await on clickThe action may not finish in orderAdd await
waitForTimeout(5000)Slow and unreliableAssert the expected state
Checks only the Save buttonDoes not prove anything was savedAssert the saved result
try and catchThe test can never fail hereRemove it

Did you catch all seven?

The most serious problems are the ones that let a broken feature pass. Here, those are the weak assertion and the swallowed failure.


📌 Prove It Can Fail

Explain Like I’m 5: Trying the Lock

Imagine: You lock the door, then push it to make sure it holds.

What it really means: Break the behaviour and confirm that the test notices.

In testing: Mutation testing automates this idea for code.

For each important test, try a controlled change, such as a wrong expected value or a mocked error response. If the test still passes, it needs work.

Our Network Mocking lesson shows how to simulate failed responses safely. For timing problems, see Flaky Test Diagnosis with AI.


🔑 Enforce What You Can Automatically

Humans should focus on judgment. Tools can catch routine problems, such as linting rules for missing await or a CI check that fails on test.only. Playwright has a forbidOnly setting:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  forbidOnly: !!process.env.CI,
});

💡 Record Your Review

Keep a short note in your pull request with what AI generated, what you changed and how you proved the test works. Reviewers then know what to focus on.


🧪 Try It Yourself

  1. Take a test you generated earlier in this level.
  2. Score it against the eight-point checklist.
  3. Run the prompt below in a new chat for a second opinion.
  4. Fix the important issues.
  5. Prove that the fixed test can fail.

📋 Ready-to-Copy Prompt

Act as a strict senior test automation reviewer. Review this Playwright
TypeScript test. Do not rewrite it.

Requirement:
[paste requirement]

Test:
[paste test]

Check purpose, requirement match, locators, assertions, waits,
independence, test data, secrets and missing await.

Return a table with: Line, Issue, Severity, Why It Matters, Suggested Fix.

Then answer: could this test pass while the feature is broken? Explain
how to prove it can fail.

⚠️ Common Mistakes

  • Approving a test because it is green
  • Judging the code only by style
  • Allowing failures to be caught and ignored
  • Accepting changed expected results without evidence
  • Reviewing only the new lines and not their effect

🎯 Key Takeaways

  • Clean-looking code can still test very little.
  • Use the same checklist on every generated test.
  • See each test fail for the right reason.
  • Automate routine checks, and keep human judgment for meaning.

🎤 Interview Questions

1. How do you review AI-generated test automation code?

Model answer: I check that it matches the requirement, then review locators, assertions, waits, independence, data and secrets. I run it and confirm it fails when the behaviour is broken.

2. What is a false positive and a false negative in test automation?

Model answer: Terminology varies, but commonly a false positive reports a failure when the product is fine, and a false negative passes when a defect exists. False negatives are dangerous because they create false confidence.

3. What is mutation testing?

Model answer: Mutation testing makes small deliberate changes to code and checks whether tests detect them. If tests still pass, they may not be protecting that behaviour.

4. Why is try and catch dangerous in a test?

Model answer: It can swallow an assertion failure, which allows the test to pass when it should fail. Error handling in tests must not hide failures that the test exists to detect.

5. What does forbidOnly do in Playwright?

Model answer: It causes the run to fail if test.only is found in the code. Teams often enable it in CI, so that a focused test cannot accidentally skip the rest of the suite.


📝 Assignment

Review five AI-generated tests with the checklist, and log each finding. Prove that at least three can fail. This completes Level 4, and you are ready to move on to testing AI products in Level 5.


❓ FAQ

Should AI review its own code?

It can give a useful second opinion, ideally in a separate session. A person must still make the final decision.

How much review is enough?

Match the review to the risk. A test covering payments needs more care than a simple display check, and every test should be seen failing once.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with How to Test AI Applications: Why AI Testing Is Different.

Flaky Test Diagnosis with AI: Triage CI Failures Faster

A test that passes and fails with no code change slowly destroys trust in your suite. AI can help you sort failures and spot patterns, but it can also suggest quick fixes that hide real bugs. In this lesson you will practise flaky test diagnosis with AI using evidence, and fix the real cause.

📖 What You’ll Learn

  • What makes a test flaky
  • How to classify a CI failure
  • What evidence to collect
  • Why retries are not a fix

💡 What Is a Flaky Test?

Explain Like I’m 5: Flaky Test

Imagine: A toy works some mornings and not others, and nobody changed it. You stop trusting it.

What it really means: A flaky test gives different results without any relevant change.

In testing: The same code passes on one run and fails on the next.

Common causeClueReal fix
TimingFails when the page is slowUse web-first assertions
Shared dataFails in parallel onlyGive each test its own data
Test orderFails after another testRemove dependencies
Unstable environmentMany unrelated failuresFix or report the environment
Unstable locatorMatches different elementsUse a more specific locator
Product race conditionReal users may see it tooReport it as a defect

🔍 Triage First

Explain Like I’m 5: Sorting Toys

Imagine: First you make piles: broken, dirty and fine. Then you know what to do with each pile.

What it really means: Triage means sorting failures before fixing them.

In testing: Decide whether it is the product, test, data or environment.

CategoryNext step
Product defectReport it, with evidence
Test defectFix the test
Test dataRepair or isolate the data
EnvironmentReport it to the owner
Flaky, cause unknownInvestigate and track it

🧩 Collect Evidence

Configure Playwright to keep useful evidence when something goes wrong:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 1 : 0,
  use: {
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
  },
});

To check whether a test is unstable, run it many times:

npx playwright test tests/example.spec.ts --repeat-each=20

Inspect failures with the Trace Viewer. A test that fails and then passes on a retry is marked flaky in the Playwright report.


🛠️ Flaky Test Diagnosis with AI

Explain Like I’m 5: Clues on the Table

Imagine: A detective lays every clue on the table before naming anyone.

What it really means: Give AI real evidence and ask for possible causes.

In testing: Share the error, step and timing details, then verify its ideas.

Good input for AI includes the exact error, the test code, how often it fails and what changed recently. Remove secrets before sharing logs.


🌍 Real-Life Example: A Fixed Wait

On our practice page, the message takes about three seconds to appear. This test waits for exactly that long, so it is a race. Sometimes the text is ready, and sometimes it is not.

import { test, expect } from '@playwright/test';

test('flaky version', async ({ page }) => {
  await page.goto('https://sqatools.in/automation-practice-page/');
  await page.getByRole('button', { name: 'Load Success' }).click();
  await page.waitForTimeout(3000);

  const message = await page.locator('#loader').textContent();
  expect(message).toBe('Data loaded successfully!');
});

The textContent call reads the value once, and that plain expect never retries. Here is the better version:

import { test, expect } from '@playwright/test';

test('stable version', async ({ page }) => {
  await page.goto('https://sqatools.in/automation-practice-page/');
  await page.getByRole('button', { name: 'Load Success' }).click();
  await expect(page.locator('#loader')).toHaveText('Data loaded successfully!');
});

The new assertion keeps checking until the text appears or the timeout ends. Learn more in Playwright Waits Explained.


🔑 Retries Are Not a Fix

A retry can hide a real bug

Retries help you collect evidence and manage temporary problems. Do not raise retries, add long waits or delete assertions just to get a green build.

If you must temporarily remove a flaky test from the main run, log an owner and a date to fix it. Otherwise, quarantined tests are easily forgotten.


🧪 Try It Yourself

  1. Put the flaky and stable tests in your project.
  2. Run both with --repeat-each=20.
  3. Open a report, and compare results.
  4. Run the prompt below on any failure you see.
  5. Confirm the suggested cause with the trace before making a change.

📋 Ready-to-Copy Prompt

You are a senior Playwright engineer helping me triage a CI failure.

Test code:
[paste test]

Exact error:
[paste error]

Failure pattern: [how often, which browser, parallel or not]
Recent changes: [changes]
Trace or log notes: [sanitised details]

Classify the failure as Product Defect, Test Defect, Test Data,
Environment or Unknown.

Rules:
- Quote the evidence for your classification.
- Give your confidence and at least one alternative explanation.
- Suggest how to confirm the cause.
- Do not suggest more retries, longer waits or weaker assertions as a fix.

⚠️ Common Mistakes

  • Rerunning until it goes green, and moving on
  • Adding waitForTimeout to hide timing problems
  • Assuming that every failure is flaky
  • Letting a failing test weaken its own assertion
  • Quarantining tests with no owner

🎯 Key Takeaways

  • A flaky test changes result with no relevant change.
  • Triage failures before you try to fix them.
  • Use evidence from traces, logs and repeated runs.
  • Retries should not replace real fixes.

🎤 Interview Questions

1. What is a flaky test, and why is it a problem?

Model answer: It is a test that gives inconsistent results without a relevant code change. It wastes time and reduces trust, so teams may begin ignoring failures, including real ones.

2. How do you investigate a flaky Playwright test?

Model answer: I run it repeatedly, then compare traces, screenshots, timing and parallel behaviour. I check for data conflicts and environment problems, then fix the root cause.

3. How do you decide if a failure is a product bug or test problem?

Model answer: I compare the actual result with the requirement, reproduce it manually if possible and review the trace and logs. If users would see the failure, it may be a product defect, even if it is intermittent.

4. Are retries good or bad?

Model answer: They are useful for handling occasional environmental problems and for capturing evidence. They become harmful when used to hide flakiness, so I still track and investigate failures that pass on retry.

5. How can AI help with failure triage?

Model answer: It can summarise logs, group similar failures and suggest possible causes. I give it sanitised evidence and require it to cite that evidence, then I confirm the cause before changing anything.


📝 Assignment

Write a test with a deliberate timing problem, then prove that it is unstable by repeating it. Fix it without adding a fixed wait. Record the error, cause, fix and evidence as a short triage note.


❓ FAQ

How many repeats prove that a test is stable?

No number gives complete proof. More repeats increase confidence, especially if you also run under realistic load and in CI.

Should flaky tests block a release?

It depends on what they cover and your team’s rules. A flaky test in a critical area needs prompt attention, as it may hide a real problem.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Review AI-Generated Automation Tests: Catch Weak Playwright Code.

Accessibility and Security Testing with AI: Safe First Steps

Two kinds of defects are easy to miss: those that block people with disabilities, and those that expose data. Testers can catch many of these early. In this lesson you will take safe first steps in accessibility and security testing with AI, and learn why tools alone can never give you the full picture.

📖 What You’ll Learn

  • What accessibility testing checks
  • How to run an automated accessibility scan
  • How AI can support basic security test ideas
  • The limits and rules you must follow

💡 What Is Accessibility Testing?

Explain Like I’m 5: Accessibility

Imagine: A building with only stairs keeps out a friend who uses a wheelchair. A ramp lets everyone in.

What it really means: Accessibility means people with different abilities can use your product.

In testing: Forms need labels, so that screen readers can announce them.

Common checks include keyboard use, form labels, colour contrast, image descriptions and clear headings. WCAG is the widely used standard.


🔍 An Automated Accessibility Scan

Explain Like I’m 5: The Spell Checker

Imagine: A spell checker finds wrong spellings. It cannot tell you whether your story makes sense.

What it really means: Automated tools find some problems, and people must check the rest.

In testing: A scan can spot a missing label, but not a confusing journey.

Install the axe integration:

npm install -D @axe-core/playwright

This test scans a page for WCAG A and AA rules and fails on serious problems.

import { test, expect } from '@playwright/test';
import AxeBuilder from '@axe-core/playwright';

test('page has no serious automatic accessibility issues', async ({ page }) => {
  await page.goto('https://sqatools.in/automation-practice-page/');

  const results = await new AxeBuilder({ page })
    .withTags(['wcag2a', 'wcag2aa'])
    // Third-party ads change on every load and are outside this page's code.
    .exclude('ins.adsbygoogle')
    .analyze();

  const serious = results.violations.filter(
    (violation) => violation.impact === 'serious' || violation.impact === 'critical',
  );

  expect(serious).toEqual([]);
});

The exclude line skips third-party ads, which can change on every page load. Only exclude content that you do not control, and say in your report what you left out. If this test fails on a practice page, read each violation as a learning exercise. Do not simply turn off the rule. If you are new to writing these tests, start with Generate Playwright Tests with AI.


🧩 Accessibility and Security Testing with AI: Where It Helps

AI can explain a violation in plain language, suggest a fix for developers and list manual checks. Then a person must check real use.

Manual checkHow to try it
Keyboard onlyUse Tab, Enter and Escape without a mouse
FocusCan you always see where you are?
Screen readerDo announcements make sense?
ZoomIs it usable at 200%?
Error messagesAre they clear and linked to fields?

🛠️ First Steps in Security Testing

Explain Like I’m 5: Checking the Doors

Imagine: You check that your own front door locks. You do not go around testing the neighbours’ doors.

What it really means: Security tests must happen only on systems you are allowed to test.

In testing: Stay within your project’s approved scope.

Get written permission

Never scan or attack a system without clear authorisation. Stay within the agreed scope, use test accounts and report findings privately through the proper process.

Testers can check many basics as part of normal work. The OWASP Top 10 is a good introduction.

Basic checkExample
AuthenticationProtected pages reject anonymous users
AuthorisationOne user cannot open another user’s order
Input handlingUnexpected text is handled safely
Sensitive dataErrors do not reveal internal details
SessionLogging out ends access

This real test checks that a protected endpoint on our practice API rejects a request that has no token.

import { test, expect } from '@playwright/test';

test('profile endpoint rejects anonymous requests', async ({ request }) => {
  const response = await request.get(
    'https://sqatools.in/wp-json/sqa-api/v1/auth/me',
  );

  expect(response.status()).toBe(401);
});

🌍 Real-Life Example: Changing an ID

A tester uses a test account to open /orders/1001. Using another test account that owns a different order, the tester tries that same URL. If the order appears, one user can see another user’s data. AI can suggest this idea, but only a real test on an approved system can confirm it.


🧪 Try It Yourself

  1. Run the accessibility scan on the practice page.
  2. Pick one violation, and use the prompt below to understand it.
  3. Try the same page using only your keyboard.
  4. Note anything that the scan did not find.
  5. Run the anonymous request test against the practice API.

📋 Ready-to-Copy Prompt

You are an accessibility specialist helping a QA engineer.

Violation from an automated scan:
[paste rule, impact, description and HTML with no personal data]

Please provide:
1. A plain-language explanation of who this affects
2. Likely fix options for a developer
3. Manual checks to confirm the problem and the fix
4. Anything an automated tool cannot confirm

Do not claim that the page is compliant with any standard.

⚠️ Common Mistakes

  • Treating a clean scan as proof of accessibility
  • Disabling rules to get a passing test
  • Scanning systems without permission
  • Using real accounts for security checks
  • Posting security issues in public channels

🎯 Key Takeaways

  • Automated scans find only some accessibility problems.
  • Keyboard and screen reader checks remain essential.
  • Security testing needs clear permission and scope.
  • Report sensitive findings through the proper process.

🎤 Interview Questions

1. What is accessibility testing?

Model answer: It checks that people with different abilities can use a product, including those using keyboards or screen readers. Teams often use WCAG as a reference, with both automated and manual checks.

2. Can automated tools fully test accessibility?

Model answer: No. They find many technical issues, such as missing labels. They cannot fully judge meaning, usability or a real screen reader experience, so manual testing is still needed.

3. What is the difference between authentication and authorisation?

Model answer: Authentication confirms who a user is. Authorisation decides what that user can access or do. A logged-in user may still be unauthorised to view another user’s data.

4. What is an insecure direct object reference?

Model answer: It occurs when an application uses an identifier from the request without properly checking permission. For example, changing an order ID could expose another user’s order.

5. How can AI support security testing responsibly?

Model answer: It can suggest test ideas and explain concepts. I use it only for approved systems, keep sensitive data out and confirm findings through authorised testing and the security team.


📝 Assignment

Run an accessibility scan and a keyboard-only review of one practice page. Then write five basic authorisation and authentication test ideas for a made-up application. Mark which ones could be automated.


❓ FAQ

Is a passing axe test enough for legal compliance?

No. It can support your work, but compliance needs a broader review. Ask qualified specialists when legal requirements apply.

Can testers do security testing without being specialists?

Yes, for basic checks in normal test scope. Deeper penetration testing needs specialist skills and explicit authorisation.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Flaky Test Diagnosis with AI: Triage CI Failures Faster.

Performance Testing with AI: Write Scripts and Read Results

A feature can work perfectly for one user and collapse with five hundred. AI can draft scripts and summarise results, but it cannot decide what “fast enough” means for your users. In this lesson you will start performance testing with AI, read the numbers correctly and stay within safe limits.

📖 What You’ll Learn

  • The main types of performance tests
  • The metrics that matter most
  • How to draft a simple k6 script
  • How to read results without being misled

💡 What Is Performance Testing?

Explain Like I’m 5: Performance Testing

Imagine: One child on a slide is easy. Thirty children at once means a long queue.

What it really means: Performance testing checks how a system behaves under load.

In testing: You measure speed, errors and stability as users increase.

Test typeQuestion it answers
LoadDoes it meet goals at expected traffic?
StressWhere does it start to fail?
SpikeCan it handle a sudden jump?
SoakDoes it stay stable for a long time?

🔍 Metrics That Matter

Explain Like I’m 5: Percentiles

Imagine: Most children finish lunch in ten minutes, but a few take an hour. The average hides those slow eaters.

What it really means: A percentile shows the time that most requests met.

In testing: p95 means 95% of requests were this fast or faster.

MetricMeaning
Response timeHow long a request takes
p95 and p99Slowest experience for most users
ThroughputRequests handled per second
Error ratePercentage of failed requests
Resource useCPU, memory and database load

Averages can hide slow requests, so use percentiles too.


🧩 Get Permission First

Never load test without permission

A load test can look like an attack and can harm real users. Run it only on systems you are authorised to test. Do not use the SQA Tools practice API or any shared demo site.


🛠️ How to Do Performance Testing with AI

Start by defining your goals with the team. AI can then help turn them into a script.

Example goal: at 5 virtual users for 30 seconds, 95% of requests finish within 800 ms, and fewer than 1% fail. These numbers are examples only.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 5,
  duration: '30s',
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<800'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/health`);

  check(res, {
    'status is 200': (r) => r.status === 200,
  });

  sleep(1);
}

Run it against your own approved test server:

k6 run -e BASE_URL=https://your-approved-test-server.example script.js

The thresholds make the run fail if your goals are not met. sleep(1) adds a pause between requests, so that each virtual user behaves more like a real one. The script is small on purpose. Start small, and raise the load only with agreement. Confirm that the endpoint works correctly first, as covered in API Testing with AI.


📌 Reading the Results

Explain Like I’m 5: Reading a Thermometer

Imagine: One number on a thermometer tells you a little. You also ask how the child feels.

What it really means: Combine several results before making a conclusion.

In testing: Compare speed, errors and server health together.

Here are made-up results from two runs.

MetricBaselineNew build
Median response180 ms190 ms
p95 response420 ms1,250 ms
Error rate0.1%0.2%

The median barely changed, but p95 is about three times slower. Some users are now having a much worse experience. A good next step is to find which requests are slow and check server metrics.

When asking AI to summarise results, give it the real numbers and require it to separate facts from guesses.


🧪 Try It Yourself

  1. Write a performance goal for one page or endpoint, using made-up numbers.
  2. Run the prompt below to draft a script.
  3. Check every URL, the load level and the thresholds.
  4. Ask a developer or trainer to approve a small run on a local test server.
  5. Compare your p95 and median results.

📋 Ready-to-Copy Prompt

You are a performance test engineer. Draft a small k6 script.

System under test: an approved test environment, using BASE_URL.
Scenario: [user journey or endpoint]
Agreed goals: [response time and error goals]
Load: [virtual users and duration]

Rules:
- Read the URL from an environment variable.
- Do not include real credentials.
- Add thresholds based only on the agreed goals.
- Add checks for the expected response.
- Keep the load exactly as stated.

After the script, explain each option and list what I should monitor
on the server while it runs.

⚠️ Common Mistakes

  • Testing a system without permission
  • Trusting only the average response time
  • Letting AI invent performance goals
  • Running from a laptop and blaming the server for local limits
  • Forgetting to compare against a baseline

🎯 Key Takeaways

  • Performance testing shows behaviour under load.
  • Use percentiles and error rates, as well as averages.
  • Agree the goals before you write the script.
  • Load test only on systems you are allowed to test.

🎤 Interview Questions

1. What is the difference between load and stress testing?

Model answer: Load testing checks behaviour at expected traffic against agreed goals. Stress testing increases traffic beyond normal levels to find where and how the system starts to fail.

2. What does p95 mean?

Model answer: It is the value below which 95% of measurements fall. A p95 of 800 ms means that 95% of requests finished within 800 ms. It reveals slow experiences that an average can hide.

3. What is a threshold in k6?

Model answer: A threshold is a pass or fail goal for a metric, such as p95 under 800 ms. If it is not met, k6 marks the run as failed, which makes it useful in a pipeline.

4. How can AI help with performance testing?

Model answer: It can draft scripts and help summarise results. I provide agreed goals and real numbers, check the load settings and confirm conclusions with server metrics and developers.

5. What would you do if p95 increased after a release?

Model answer: I would confirm that the test conditions matched and repeat the run. Then I would find the slow requests and review server, database and dependency metrics to help locate the cause.


📝 Assignment

Create a performance test plan with a goal, a load model, metrics and a stop condition. Have AI draft the script, and record every setting you changed. Run it only on a local or approved test system.


❓ FAQ

Do I need k6 specifically?

No. JMeter, Gatling and other tools work too. The concepts remain the same.

Can Playwright be used for load testing?

It is built for browser testing and uses more resources per user. Dedicated load tools are normally a better choice, though browser timings can complement them. See Playwright with Docker for running tests at scale.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Accessibility and Security Testing with AI: Safe First Steps.

SQL Database Validation with AI: Check Data Like a Tester

A screen can say “Order saved” while the database holds the wrong data. Testers who can check the data find problems others miss. In this lesson you will practise SQL database validation with AI, write read-only checks and learn how to confirm that a query is really correct.

📖 What You’ll Learn

  • Why testers validate data in the database
  • Three common data quality checks
  • How to give AI a safe database description
  • How to verify a generated query

💡 Why Check the Database?

Explain Like I’m 5: Checking the Piggy Bank

Imagine: The app says you saved ten coins. You open the piggy bank and count them.

What it really means: Database checks confirm what was really stored.

In testing: The screen can show success while the data is wrong.


🔍 Describe the Tables, Not the Data

Explain Like I’m 5: A Map of the Toy Room

Imagine: You can tell a friend where the toy shelves are without handing over your toys.

What it really means: Give AI table and column names, not real records.

In testing: A schema is usually enough to write a query.

Our example uses two tables. Use made-up data only.

CREATE TABLE customers (
  id INTEGER PRIMARY KEY,
  email TEXT NOT NULL
);

CREATE TABLE orders (
  id INTEGER PRIMARY KEY,
  customer_id INTEGER NOT NULL,
  total DECIMAL(10, 2) NOT NULL,
  status TEXT NOT NULL
);

INSERT INTO customers (id, email) VALUES
  (1, 'asha@example.com'),
  (2, 'ravi@example.com'),
  (3, 'asha@example.com');

INSERT INTO orders (id, customer_id, total, status) VALUES
  (101, 1, 500.00, 'PAID'),
  (102, 2, -50.00, 'PAID'),
  (103, 9, 300.00, 'PAID');

🧩 Three Useful Validation Queries

Explain Like I’m 5: Finding Odd Socks

Imagine: You look for a sock with no pair, two socks that are the same and a sock with a hole.

What it really means: Look for missing links, duplicates and invalid values.

In testing: Each query should return no rows when the data is healthy.

Orders that have no matching customer:

SELECT o.id, o.customer_id
FROM orders o
LEFT JOIN customers c ON c.id = o.customer_id
WHERE c.id IS NULL;

This returns order 103, because customer 9 does not exist.

Email addresses used more than once:

SELECT email, COUNT(*) AS total
FROM customers
GROUP BY email
HAVING COUNT(*) > 1;

This returns asha@example.com with a total of 2.

Paid orders with an invalid total:

SELECT id, total
FROM orders
WHERE status = 'PAID' AND total <= 0;

This returns order 102, with a total of -50.00.

A helpful pattern is to write checks so that zero rows means healthy. Any returned row is then something to investigate. Learn more in our SQL Tutorial and SQL Joins lessons.


🛠️ How to Do SQL Database Validation with AI

Give the AI the schema, the database type and the rule you want to check. Then verify its query using data where you already know the answer.

CheckWhat to do
CorrectnessCreate rows that should and should not match
DialectConfirm the syntax for your database
JoinsCheck whether rows are lost or duplicated
NULL valuesTest how missing values behave
SafetyConfirm that the query only reads data

Run only read-only queries on shared systems

Never run UPDATE, DELETE or DROP statements suggested by AI on a shared database. Use a read-only account, and work on an approved test environment.


🌍 Real-Life Example: A Join That Lies

A tester asks AI to count paid orders per customer. The query joins orders to payments, but an order with two payment attempts is counted twice. The number looks believable. Testing the query with one known order and two payment rows exposes the mistake. Always test a query with data where you know the right answer.


🧪 Try It Yourself

  1. Create the tables above in a local test database.
  2. Run the three queries, and compare the results.
  3. Add one more bad row of your own.
  4. Ask AI for a new check, using the prompt below.
  5. Add rows that prove the query works.

📋 Ready-to-Copy Prompt

You are a senior QA engineer writing SQL data validation checks.

Database type: [MySQL, PostgreSQL or other]
Tables and columns:
[paste schema with no real data]

Business rule to validate:
[describe the rule]

Write one read-only SELECT query that returns only rows breaking the
rule, so that zero rows means healthy.

Rules:
- Do not use INSERT, UPDATE, DELETE, DROP or ALTER.
- Use only the tables and columns above.
- Explain the query in plain language.
- Give me 2 test rows that should be returned and 2 that should not.

⚠️ Common Mistakes

  • Pasting real customer records into an AI tool
  • Running AI-generated changes on shared databases
  • Trusting a query because it runs without an error
  • Forgetting how NULL values behave
  • Ignoring duplicated rows caused by joins

🎯 Key Takeaways

  • Database checks confirm what was really saved.
  • Share the schema with AI, and keep real data out.
  • Design validation queries so that zero rows means healthy.
  • Prove each query using data with a known answer.

🎤 Interview Questions

1. Why would a tester validate data in the database?

Model answer: The UI can appear correct while the stored data is wrong or incomplete. Database checks confirm correct saving, relationships and calculations, and they are useful for investigating defects.

2. How would you find records with no matching parent?

Model answer: I would use a LEFT JOIN from the child table to the parent table and filter where the parent key is NULL. Those rows have no matching parent record.

3. How do you find duplicate values in a column?

Model answer: I group by the column and use HAVING with COUNT greater than one. Any returned value appears more than once.

4. How do you verify an AI-generated SQL query?

Model answer: I check the schema, read the logic and run it on small test data where I know the answer. I include rows that should and should not match, and I look carefully at joins and NULL handling.

5. How do you stay safe when using AI with databases?

Model answer: I share only the schema, use approved tools and run read-only queries through limited accounts on test environments. I would never run generated changes on shared data without review and approval.


📝 Assignment

Write five read-only validation checks for a small made-up shop database. For each one, create rows that prove it finds a problem and rows that prove it ignores healthy data. Try the QA Test Results Tracker project for more practice.


❓ FAQ

Do manual testers need SQL?

It is not always required, but it is a valuable skill. Even simple read-only queries can help you confirm results and investigate defects.

Can AI connect directly to my database?

Some tools can. Keep them away from production and use limited read-only access on approved test systems.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Performance Testing with AI: Write Scripts and Read Results.