LLM Regression Testing in CI: Evaluation Tools That Catch Drift

A small prompt edit or a new model version can quietly change how an AI feature behaves. Without checks, you may only find out from unhappy users. In this final lesson of Level 5 you will set up LLM regression testing in CI, so every important change is compared with a trusted baseline.

---

💡 Why AI Features Regress

Change	Possible effect
Prompt edit	Different tone or missing rules
New model version	Different accuracy and refusals
Updated documents	Better or worse retrieval
Temperature or settings	More variation
New guardrail	Too many refusals

Drift means quality changes over time, even if your own code did not.

---

🔍 LLM Regression Testing with a Baseline

This Python example flags any category that dropped by more than the allowed amount. The scores are made up.

baseline = {"returns": 0.96, "delivery": 0.92, "safety": 1.00}
current = {"returns": 0.97, "delivery": 0.84, "safety": 1.00}
ALLOWED_DROP = 0.02

failed = [
    name for name in baseline
    if current[name] < baseline[name] - ALLOWED_DROP
]

print("Failed categories:", failed)

Failed categories: ['delivery']

The overall average would hide some of this, but a category comparison shows exactly where quality fell.

---

🧩 A Simple Evaluation Tool

Popular choices include promptfoo, DeepEval and Ragas. Here is a small promptfoo config. Replace the provider with one you are approved to use, and check the promptfoo documentation for current options.

description: Returns assistant regression
prompts:
  - file://prompts/returns-assistant.txt
providers:
  - id: your-provider:your-model
tests:
  - description: Return period is correct
    vars:
      question: How many days can I return a shirt?
    assert:
      - type: icontains
        value: 30 days
      - type: llm-rubric
        value: The answer is polite and does not promise refunds for used items.
  - description: Unknown policy is not invented
    vars:
      question: Can I return a product after 10 years?
    assert:
      - type: not-icontains
        value: yes, you can

Run it with:

npx promptfoo@latest eval -c promptfooconfig.yaml

---

🛠️ Running in CI

Run the evaluation when prompts or AI configuration change. This GitHub Actions example uses a repository secret. The name of the key depends on your provider.

name: LLM regression
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npx promptfoo@latest eval -c promptfooconfig.yaml
        env:
          PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY }}
          PROMPTFOO_PASS_RATE_THRESHOLD: 95

Promptfoo exits with an error code when the pass rate falls below that threshold. Our GitHub Actions Tutorial explains workflows in depth.

---

📌 Design a Sensible Gate

Check	Suggested handling
Critical safety cases	Must all pass
High-risk knowledge	Must meet the agreed rate
Medium quality	Warn, and track the trend
Flaky AI judge results	Repeat, and send to a person
Latency and cost	Alert when budgets are exceeded

---

🔑 After Release

Offline tests cannot predict everything. Monitor real use safely, review a sample and add new failures to your golden dataset.

---

🌍 Real-Life Example: A Silent Model Update

A provider updates a model. No code changes, but the chatbot becomes more cautious and refuses normal delivery questions. A scheduled weekly evaluation catches the rise in refusals before customers complain, and the team pins the earlier version while it investigates.

---

🧪 Try It Yourself

1. Change the scores in the Python example so that safety falls.
2. Decide if safety should allow any drop at all.
3. Create a small config with three test cases for a made-up bot.
4. Use the prompt below to review your gate.
5. If you have an approved provider, run the evaluation.

---

📋 Ready-to-Copy Prompt

You are a senior QA engineer designing an LLM regression process.

Feature: [feature]
Risk categories: [categories]
What can change: [prompts, models, documents, settings]
CI tool: [tool]
Budget limits: [limits]

Design a regression plan with: Trigger, Dataset Used, Checks, Pass or
Warn Rule, Who Reviews Failures and Where Results Are Stored.

Rules:
- Do not invent thresholds. Write "[TEAM TO SET]" for each number.
- Keep safety rules separate from average quality scores.
- Include a way to handle variation between runs.

---

⚠️ Common Mistakes

• Evaluating only when someone remembers
• Comparing runs that used different data
• Using one score as the gate
• Blocking releases on an unreliable AI judge
• Letting a model update without a re-run

---

🎯 Key Takeaways

• Prompts, models and data can all change quality.
• Compare each change with an approved baseline.
• Gate strictly on safety, and track lower risks.
• Keep monitoring real use after release.

---

🎤 Interview Questions

1. What is LLM regression testing?

Model answer: It reruns a trusted evaluation set after changes to prompts, models, data or settings and compares results with a baseline. This reveals where quality or safety has changed.

2. How would you integrate LLM evaluation into CI?

Model answer: I would trigger a small, risk-focused set when AI configuration changes, store results and apply clear gates. A larger evaluation can run on a schedule or before release.

3. How do you handle non-deterministic results in CI?

Model answer: I rely on rule-based checks where possible, repeat important cases and compare results across categories. Borderline or judge-based failures can go to human review instead of immediately blocking.

4. What is model drift?

Model answer: It is a change in AI behaviour or quality over time. It can result from model updates, changing data or new user behaviour. Scheduled evaluation and monitoring help detect it.

5. How do you set a quality gate for an AI feature?

Model answer: I base it on risk. Critical safety cases must pass, high-risk categories must meet agreed targets and lower-risk measures may produce warnings. I also consider latency, cost and sample size.

---

📝 Assignment

Create a regression plan for a made-up chatbot, including a baseline, categories, triggers and gates. Write a config with at least five cases. This completes Level 5, and you are ready to build your own QA agent in Level 6.

---

❓ FAQ

Which evaluation tool should I choose?

Choose one that fits your language, stack and security rules. Try a small proof of concept using your own test cases.

Should AI evaluation block a release?

Critical checks can block. Less reliable measures may be better as warnings for a person to review.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent vs Chatbot vs Automation Script: What Testers Should Know.