Prompt Versioning and Evaluation Gates: CI/CD for AI Agents

A single changed sentence in a prompt can make your agent worse, and no unit test will notice. Prompt versioning treats prompts as code, so each change is reviewed and traceable. In this lesson you will add an evaluation gate that blocks bad releases, and a release pipeline that a person must approve.

---

💡 Why Prompt Versioning Matters

Your prompts already have a version field, and they live in Git. Follow two simple rules:

1. Change the version number whenever you change a prompt’s meaning.
2. Change prompts through pull requests, just like code.

---

🔍 What Is an Agent Release?

Part	Tracked by
Application code	Git commit and image tag
Prompts	Version number and Git history
Model	LLM_MODEL setting
Dependencies	Lockfile
Tool definitions	Pinned package version

Run the evaluation pack whenever any of these change.

---

🧩 Build the Evaluation Gate

Create src/evaluation/gate.ts.

export type EvalSummary = {
  release: string;
  passRate: number;
  safetyFailures: number;
  averageCostUsd: number;
};

export type GateLimits = { maxPassRateDrop: number; maxCostIncreasePercent: number };

export function evaluateGate(
  baseline: EvalSummary,
  candidate: EvalSummary,
  limits: GateLimits = { maxPassRateDrop: 5, maxCostIncreasePercent: 25 },
): { pass: boolean; reasons: string[] } {
  const reasons: string[] = [];

  if (candidate.safetyFailures > 0) {
    reasons.push(`${candidate.safetyFailures} safety failure(s)`);
  }
  const drop = baseline.passRate - candidate.passRate;
  if (drop > limits.maxPassRateDrop) {
    reasons.push(`Pass rate dropped by ${drop} points`);
  }
  if (baseline.averageCostUsd > 0) {
    const increase = (candidate.averageCostUsd / baseline.averageCostUsd - 1) * 100;
    if (increase > limits.maxCostIncreasePercent) {
      reasons.push(`Cost rose by ${Math.round(increase)}%`);
    }
  }
  return { pass: reasons.length === 0, reasons };
}

Create tests/unit/gate.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { evaluateGate } from '../../src/evaluation/gate.js';

const baseline = { release: 'r1', passRate: 90, safetyFailures: 0, averageCostUsd: 0.2 };

test('an equal or better release passes', () => {
  assert.equal(evaluateGate(baseline, { ...baseline, release: 'r2', passRate: 92 }).pass, true);
});

test('any safety failure blocks the release', () => {
  const result = evaluateGate(baseline, { ...baseline, passRate: 100, safetyFailures: 1 });
  assert.deepEqual(result, { pass: false, reasons: ['1 safety failure(s)'] });
});

test('a large drop in quality or rise in cost blocks the release', () => {
  assert.equal(evaluateGate(baseline, { ...baseline, passRate: 80 }).pass, false);
  assert.equal(evaluateGate(baseline, { ...baseline, averageCostUsd: 0.3 }).pass, false);
});

Create scripts/eval-gate.ts, so that a pipeline can run the check.

import { readFile } from 'node:fs/promises';
import { evaluateGate, type EvalSummary } from '../src/evaluation/gate.js';

const read = async (path: string) => JSON.parse(await readFile(path, 'utf8')) as EvalSummary;
const [baselinePath, candidatePath] = process.argv.slice(2);

const result = evaluateGate(await read(baselinePath), await read(candidatePath));
console.log(result.pass ? 'Evaluation gate passed' : `Evaluation gate failed: ${result.reasons.join('; ')}`);
process.exit(result.pass ? 0 : 1);

Each file contains a summary of scored results, like this one.

{
  "release": "2026-11-10-prompt-v2",
  "passRate": 90,
  "safetyFailures": 0,
  "averageCostUsd": 0.2
}

Store evaluation/baseline.json for your current release. Because people score your evaluation pack, produce evaluation/candidate.json from that reviewed work, and do not generate it with a model.

---

🛠️ Create the Release Pipeline

Create .github/workflows/release.yml.

name: Release

on:
  workflow_dispatch:

permissions:
  contents: read
  packages: write

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test:unit
      # evaluation/candidate.json comes from your reviewed evaluation run.
      - run: npx tsx scripts/eval-gate.ts evaluation/baseline.json evaluation/candidate.json

  publish:
    needs: evaluate
    runs-on: ubuntu-latest
    # Set required reviewers on this environment in your repository settings.
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

Step	Purpose
workflow_dispatch	A person starts each release
evaluate job	Code checks and the evaluation gate
needs: evaluate	No image is built if the gate fails
environment: production	Requires your named reviewers
Tag with github.sha	Every image traces back to a commit

In your repository settings, add required reviewers to the production environment. Your team’s policy may also require pinning third-party actions to a full commit hash.

---

📌 Roll Back Safely

What changed	How to roll back
Code or prompts	Redeploy the previous image
Model	Restore the previous LLM_MODEL
Database	Keep changes backward compatible

Make database changes in two steps: add new columns first, and remove old ones in a later release. An older image can then still run. You learned to compare results over time in LLM Regression Testing in CI.

---

🌍 Real-Life Example: The Friendly Rewrite

Someone rewrites a planner prompt to be “friendlier”. The pass rate stays the same, but a safety case now fails, because the planner no longer refuses invented requirements. The gate blocks the release, and the team finds the sentence that caused it.

---

🧪 Try It Yourself

1. Create the files and run the unit tests.
2. Create two summary files, and run the gate.
3. Add a safety failure, and confirm that the script exits with 1.
4. Change a prompt, and increase its version.
5. Write down what a rollback would look like.

---

📋 Ready-to-Copy Prompt

You are a release manager reviewing a change to an AI agent.

What changed: [prompt, model, code or tools]
Baseline results: [paste summary]
Candidate results: [paste summary]
Individual failures: [paste with no secrets]

Tell me whether you would release it, what looks risky even if the numbers
pass and what I should check by hand. Do not change any numbers.

---

⚠️ Common Mistakes

• Editing prompts directly on a server
• Forgetting that a model change is a release
• Releasing when an evaluation has not run
• Using an average to hide a safety failure
• Making database changes that cannot be rolled back

---

🎯 Key Takeaways

• Prompts are code, so version and review them.
• A release includes code, prompts, model and tools.
• Automatic gates block known regressions.
• Practise rolling back before you need to.

---

🎤 Interview Questions

1. How do you version prompts?

Model answer: I store them in Git with a version number and change them through pull requests. Each model call logs the version, so results stay traceable.

2. What is an evaluation gate?

Model answer: It is an automatic check that compares a candidate release with a baseline. It blocks the release if quality falls, cost rises too far or there is a safety failure.

3. Why treat a model change as a release?

Model answer: A different model can behave differently with the same prompt. It needs the same evaluation as a code change.

4. How do you roll back an AI agent?

Model answer: I redeploy the previous image and restore its model setting. Backward-compatible database changes make that safe.

5. Should a gate replace human review?

Model answer: No. It catches known regressions, while people look at individual failures and at things the metrics do not capture.

---

📝 Assignment

Create baseline and candidate summaries from your evaluation pack. Add a rule that blocks a release if average review time rises by more than 20%, and add a unit test.

---

❓ FAQ

Should prompts live in a separate tool?

Git is a good start. A prompt management tool can help when non-engineers need to edit prompts.

How often should I run the evaluation?

Run it for each release candidate, and on a schedule, because a provider can change a model’s behaviour.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Observability: Logs, LLM Traces and Alerts.