LLM Evaluation Metrics for Testers: Faithfulness, Relevance and Hallucination Rate

“The chatbot seems good” is not a measurement. Numbers let you compare versions and spot problems, but only if you understand what they mean. In this lesson you will learn the main LLM evaluation metrics for testers, calculate them simply and avoid being misled by one neat score.

---

💡 The Main LLM Evaluation Metrics

Metric	Question it answers	Example failure
Relevance	Did it answer the question asked?	Talks about delivery when asked about returns
Faithfulness	Is it supported by the source?	Adds a rule that is not in the policy
Correctness	Does it match the expected answer?	Says 60 days instead of 30
Hallucination rate	How often does it make things up?	Invents an order number
Refusal accuracy	Does it refuse the right things?	Refuses a normal question
Latency and cost	Is it fast and affordable?	Too slow or expensive at scale

---

🔍 Faithful Is Not the Same as Correct

---

🧩 A Simple Calculation

Here, a reviewer has labelled five answers. This is made-up data.

reviews = [
    {"relevant": True, "faithful": True},
    {"relevant": True, "faithful": False},
    {"relevant": False, "faithful": True},
    {"relevant": True, "faithful": True},
    {"relevant": True, "faithful": True},
]

total = len(reviews)
relevance = sum(r["relevant"] for r in reviews) / total
faithfulness = sum(r["faithful"] for r in reviews) / total
hallucination_rate = 1 - faithfulness

print(f"Relevance: {relevance:.0%}")
print(f"Faithfulness: {faithfulness:.0%}")
print(f"Hallucination rate: {hallucination_rate:.0%}")

Relevance: 80%
Faithfulness: 80%
Hallucination rate: 20%

Five answers are too few to trust, so this only demonstrates the arithmetic. Define your own rules for what counts as a hallucination, and use them consistently.

---

🛠️ AI Judges

An AI judge is a model that scores another model’s answers against written criteria. It is fast and handles meaning, but it can be wrong, inconsistent or too generous.

To check a judge:

1. Label a sample of answers yourself.
2. Run the judge on the same sample.
3. Count where you disagree.
4. Improve the rubric, or add human review.
5. Check it again when the judge model changes.

---

📌 Do Not Trust One Number

A single score can hide serious problems.

Category	Pass rate	Cases
Delivery	98%	100
Returns	94%	100
Privacy	60%	5

These numbers are made up. The overall pass rate is about 95%, which looks strong. Privacy has only 5 cases, so its low score barely moves the average. Always report by category and by risk, and show the number of cases.

---

🌍 Real-Life Example: A Fake Improvement

A team changes a prompt, and the average score rises by two points. Looking closer, the model has started refusing difficult questions. Refused answers avoid hallucinations, so faithfulness goes up. But users now get less help. Tracking refusal accuracy reveals the trade-off.

---

🧪 Try It Yourself

1. Use three answers from a chatbot, or write made-up ones.
2. Label each for relevance and faithfulness by hand.
3. Run the prompt below as an AI judge.
4. Note every place where you disagree.
5. Improve the rubric, and try again.

---

📋 Ready-to-Copy Prompt

You are evaluating a chatbot answer. Use only the source and rubric below.

Question: [question]
Source text: [approved source]
Answer: [answer to evaluate]

Rubric:
- Relevant: the answer addresses the question asked.
- Faithful: every factual claim is supported by the source.

Return JSON with these fields: relevant (true or false), faithful
(true or false), unsupported_claims (list), reason (one sentence).

If the source does not contain enough information, say so in the reason.
Do not use outside knowledge.

---

⚠️ Common Mistakes

• Reporting one average score
• Assuming a higher score always means a better product
• Trusting an AI judge without checking it
• Comparing runs that used different datasets
• Ignoring the number of test cases

---

🎯 Key Takeaways

• Use several metrics, because each measures something different.
• Faithful answers can still be wrong if the source is wrong.
• Check AI judges against human labels.
• Report by category, and show your sample size.

---

🎤 Interview Questions

1. What is the difference between faithfulness and relevance?

Model answer: Faithfulness checks whether claims are supported by the provided source. Relevance checks whether the response addresses the user’s question. An answer can be faithful but irrelevant, or relevant but unsupported.

2. How would you measure hallucination rate?

Model answer: I would define what counts as unsupported, then evaluate a representative set of answers using humans or a validated judge. The rate is the share containing unsupported claims, reported by category.

3. What is an LLM judge, and what are its limits?

Model answer: It is a model that scores outputs against criteria. It scales well, but it can be inconsistent or biased. I compare it with human labels and review risky or uncertain results.

4. Why can one evaluation score be misleading?

Model answer: An average can hide failures in small but critical categories. It can also improve for the wrong reason, such as refusing more questions. I report by category, risk and sample size.

5. How do you compare two prompt versions fairly?

Model answer: I use the same dataset, settings and judge, and repeat runs where variation matters. I compare by category and check trade-offs such as accuracy, refusals, latency and cost.

---

📝 Assignment

Evaluate 15 answers by hand, then with an AI judge. Calculate how often you agreed and report results by category. Write three sentences explaining what the numbers do and do not prove. Review AI hallucinations to decide what counts as one, and use your golden dataset as input.

---

❓ FAQ

Which metric is most important?

It depends on the product and its risks. For a policy chatbot, correctness and faithfulness are often top priorities.

Do I need a paid tool to calculate these?

No. You can start with human labels and a spreadsheet, and add tools as your needs grow.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Testing RAG Applications: Retrieval, Grounding and Answers.