Testing RAG Applications: Retrieval, Grounding and Answers

Many company chatbots answer from their own documents. That design is called RAG. When its answer is wrong, the fault could be in the documents, the search or the model. In this lesson you will learn testing RAG applications one stage at a time, so you can find where the problem really is.

---

💡 What Is RAG?

RAG stands for Retrieval-Augmented Generation. It has three main stages:

Stage	What happens	What can go wrong
Documents	Content is prepared and split into chunks	Old, missing or badly split content
Retrieval	The system finds matching chunks	Finds the wrong chunks, or misses the right one
Generation	The model writes an answer from them	Ignores the source or adds to it

---

🔍 Testing RAG Applications One Stage at a Time

---

🧩 Test the Documents

Check that the source content is correct, current and approved. Check that private documents are not indexed for the wrong users. A perfect system still gives a wrong answer if its source is out of date.

---

🛠️ Test Retrieval

For each test question, a person lists which documents should be found. You can then measure the search on its own.

retrieved = ["doc3", "doc7", "doc1", "doc9", "doc2"]
relevant = {"doc1", "doc3", "doc5"}
k = 3

top_k = retrieved[:k]
hits = [doc for doc in top_k if doc in relevant]

precision = len(hits) / k
recall = len(hits) / len(relevant)

print(f"Hits in top {k}: {hits}")
print(f"Precision@{k}: {precision:.2f}")
print(f"Recall@{k}: {recall:.2f}")

Hits in top 3: ['doc3', 'doc1']
Precision@3: 0.67
Recall@3: 0.67

Two of the top three results are useful, and two of the three relevant documents were found. doc5 was missed, so an answer that needs it may be incomplete.

---

📌 Test the Generated Answer

Common checks include:

Check	Question
Faithfulness	Is every claim supported by the retrieved text?
Relevance	Does it answer the question?
Unknown handling	Does it admit when the documents lack the answer?
Citations	Do the links really support the claim?
Conflicts	What happens when two documents disagree?

Learn the scoring ideas in LLM Evaluation Metrics for Testers.

---

🔑 Diagnose the Failure

Right document found?	Answer correct?	Likely problem
No	No	Retrieval or document content
Yes	No	Generation or prompt
No	Yes	Lucky, or model used outside knowledge
Yes	Yes	Working as intended

The third row deserves attention. A correct answer without the right source may mean that the model used its general knowledge, which it should not do for company policy.

---

🌍 Real-Life Example: The Old Policy

A chatbot says returns are accepted for 30 days, but the current policy is 14 days. The tester checks what was retrieved, and finds the old PDF ranked first. The model was faithful to its source. The real fix is to remove the old document, and add a test for that question.

---

🧪 Try It Yourself

1. Write three short made-up policy documents.
2. Write five questions, and list which document answers each.
3. Pretend a search returned certain documents, and calculate precision and recall.
4. Use the prompt below to create test questions.
5. Add one question that the documents cannot answer.

---

📋 Ready-to-Copy Prompt

You are a QA engineer testing a RAG application.

Approved documents:
[paste labelled documents with no private data]

Create a test table with: Question, Expected Document IDs, Required
Facts, Must Not Claim and Test Type.

Include direct questions, reworded questions, questions needing two
documents, questions with conflicting sources, and questions the
documents cannot answer.

Rules:
- Use only the documents above.
- For an unanswerable question, set the expected behaviour to
  "says the information is not available".

---

⚠️ Common Mistakes

• Evaluating only the final answer
• Forgetting to test outdated documents
• Assuming a cited source supports the claim
• Skipping questions that have no answer
• Forgetting document access rules

---

🎯 Key Takeaways

• RAG has document, retrieval and generation stages.
• Test each stage to find the true fault.
• Precision checks for junk, and recall checks for gaps.
• Retrieval must respect user access rules.

---

🎤 Interview Questions

1. What is RAG?

Model answer: RAG stands for Retrieval-Augmented Generation. The system finds relevant content from a knowledge base and gives it to the model, so that the answer can be based on that information.

2. How would you test a RAG application?

Model answer: I would test the documents, retrieval and generation separately. I measure whether the expected chunks are found, whether answers are supported and relevant, and how the system handles unknown or conflicting information.

3. What is the difference between precision and recall in retrieval?

Model answer: Precision is the share of retrieved items that are relevant. Recall is the share of all relevant items that were retrieved. Together, they show junk and missing content.

4. If a RAG answer is wrong, how do you find the cause?

Model answer: I inspect the retrieved chunks first. If the right content was missing, I investigate documents or retrieval. If it was present and the answer was still wrong, I examine the prompt and generation.

5. What security risks are specific to RAG?

Model answer: Retrieval may expose documents that a user should not access, and retrieved content can contain malicious instructions. I test access controls and treat document text as untrusted.

---

📝 Assignment

Build a small RAG test plan with ten questions and expected documents. Calculate precision and recall for at least three made-up searches. Include two access-control tests and one prompt injection test inside a document.

---

❓ FAQ

What is a chunk?

A chunk is a small section of a document that is stored and searched. Poor chunking can split important information apart.

Do I need access to the vector database?

It helps to see what was retrieved. Ask developers to log retrieved document IDs in your test environment.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Testing AI Agents: Tool Calls, Loops and Stop Conditions.