Golden Dataset for LLM Evaluation: Build Your Test Set

You cannot measure an AI feature without good examples. A golden dataset is a trusted set of test cases that you run again and again. In this lesson you will build a golden dataset for LLM evaluation, cover the cases that matter and keep it trustworthy over time.

---

💡 What Is a Golden Dataset?

An AI answer can vary, so “expected” often means required facts and rules. It does not have to mean exact words.

---

🔍 What Goes in Each Case

Field	Purpose	Example
ID	Tracks the case	R1
Input	What the user says	How many days can I return a shirt?
Must include	Facts required	30 days
Must not include	Forbidden claims	any time
Category	Groups results	returns
Risk	Sets strictness	high
Source	Proves the answer	Returns policy, section 2

---

🧩 Golden Dataset Coverage

Case type	Example
Common questions	The questions users ask most
Hard wording	Typing errors and vague phrasing
Edge cases	“I bought it exactly 30 days ago”
Unknown answers	Questions the sources cannot answer
Out of scope	“What is the weather today?”
Safety	Requests for private data
Real failures	Bad answers from production

---

🛠️ Check the Dataset Itself

A dataset can have bugs too. This Python script checks that every case has its required fields, and shows the balance of categories and risks. It uses made-up data.

from collections import Counter

golden_set = [
    {"id": "R1", "question": "How many days can I return a shirt?",
     "must_include": ["30 days"], "category": "returns", "risk": "high"},
    {"id": "R2", "question": "Can I return a used item?",
     "must_include": ["unused"], "category": "returns", "risk": "high"},
    {"id": "D1", "question": "Do you deliver on Sundays?",
     "must_include": ["no"], "category": "delivery", "risk": "medium"},
    {"id": "O1", "question": "What is the weather today?",
     "must_include": ["cannot help"], "category": "out_of_scope", "risk": "low"},
]

required = {"id", "question", "must_include", "category", "risk"}
for case in golden_set:
    missing = required - case.keys()
    assert not missing, f"{case['id']} is missing {sorted(missing)}"

print(Counter(case["category"] for case in golden_set))
print(Counter(case["risk"] for case in golden_set))

Counter({'returns': 2, 'delivery': 1, 'out_of_scope': 1})
Counter({'high': 2, 'medium': 1, 'low': 1})

This simple check shows immediately if a category has too few examples.

---

📌 How AI Can Help

AI is useful for creating different wordings of the same question. But if a model writes both the questions and the answers, it may repeat its own mistakes. Base expected answers on approved sources, and have a subject expert review them. The checks in How to Review AI-Generated Test Cases work well here too.

---

🔑 Keep It Fresh

• Add every important production failure as a new case.
• Update expected answers when a policy changes.
• Remove cases that no longer apply.
• Record who approved each change, and when.
• Keep the file under version control.

---

🌍 Real-Life Example: A Stale Answer

A company changes its returns period from 30 days to 14 days. The bot is updated correctly, but the golden dataset still expects 30. Suddenly, correct answers are marked as failures. A tester checks the source and updates the dataset. Without a named owner, the team might have rolled back a correct change.

---

🧪 Try It Yourself

1. Write a small policy of five rules for a made-up shop.
2. Run the prompt below to draft 12 cases.
3. Check every case against your policy.
4. Add one case that the AI missed.
5. Run the Python check above using your cases.

---

📋 Ready-to-Copy Prompt

You are helping me build a golden dataset for evaluating a chatbot.

Approved source text:
[paste approved text with no personal data]

Create [number] test cases as a table with: ID, User Input, Must Include,
Must Not Include, Category, Risk and Source Quote.

Include common, vague, edge-case, unknown-answer, out-of-scope and safety
cases.

Rules:
- Base every expected answer on the source text.
- If the source cannot answer, set Must Include to "says it does not know".
- Quote the source line for each case.
- Do not invent policy details.

---

⚠️ Common Mistakes

• Including only easy, happy-path questions
• Letting AI write unreviewed expected answers
• Forgetting to update the set when policies change
• Using private customer conversations carelessly
• Measuring only an overall score without categories

---

🎯 Key Takeaways

• A golden dataset is a trusted, repeatable test set.
• Expected results are often facts and rules, not exact words.
• Cover common, edge, unknown and unsafe cases.
• Review and update the dataset like any test asset.

---

🎤 Interview Questions

1. What is a golden dataset?

Model answer: It is a curated set of inputs with trusted expected outcomes used to evaluate an AI system repeatedly. It lets a team compare versions and detect regressions.

2. What makes a good LLM evaluation dataset?

Model answer: It represents real use and covers common, difficult, unknown, out-of-scope and safety cases. Expected outcomes are reviewed by experts, linked to sources and kept up to date.

3. Can you use AI to generate an evaluation dataset?

Model answer: Yes, especially for variations in wording. I base it on approved sources and have people review expected outcomes, because a model can introduce errors or shared blind spots.

4. How do you maintain a golden dataset?

Model answer: I version it, assign an owner and add real production failures. I update it when policies change and remove obsolete cases, while recording why each change was made.

5. Why should you report evaluation results by category?

Model answer: An overall score can hide a serious weakness. A chatbot may score well overall but fail safety or billing questions, so category results show where the real risk lies.

---

📝 Assignment

Build a golden dataset of 25 cases for a made-up support bot. Include at least five edge cases and three unknown-answer cases. Have someone else review ten expected answers, and record what changed. Keep it in Git, as you learned in our Git and GitHub Tutorial.

---

❓ FAQ

How big should my dataset be?

Start small with your most important cases, then expand. Quality and coverage matter more than a large number.

Should I use the same data to improve prompts and to evaluate them?

Keep a separate set for final checks. Otherwise, a prompt can become tuned to your test questions and appear better than it really is.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Evaluation Metrics for Testers: Faithfulness, Relevance and Hallucination Rate.