AI Agent CI Pipeline: GitHub Actions and Quality Metrics

A project that runs only on your laptop is not finished. An AI agent CI pipeline checks every change automatically. The most important design decision is what to leave out: the model. In this lesson you will build the pipeline, then add metrics that show whether your agent is really getting better.

---

💡 Design the AI Agent CI Pipeline

Runs on every change	Runs separately, with approval
Lint and type check	Generating new tests with a model
Unit tests with a mock	Real model evaluation runs
A small Playwright smoke test	Any healing work

Model calls cost money, need secrets and vary between runs. Your mock-based tests give fast and repeatable results.

---

🔍 Create the Workflow

Create .github/workflows/ci.yml.

name: CI

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read

jobs:
  checks:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      BASE_URL: http://127.0.0.1:4321
      HEADLESS: 'true'
      TEST_USER_EMAIL: demo@example.test
      TEST_USER_PASSWORD: Demo-Pass-123
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test:unit
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test tests/smoke.spec.ts
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: |
            playwright-report/
            artifacts/test-results/
          retention-days: 14

Choice	Reason
permissions: contents: read	The job gets the least access it needs
timeout-minutes	A hung run cannot continue forever
npm ci	Installs the exact locked versions
!cancelled()	Evidence is kept even after a failure
Synthetic values in env	No real secret is needed

The email and password above belong only to the local demo, so they are safe here. Real credentials must go in your CI secret store. For more, see Playwright GitHub Actions.

---

🧩 Measure What Matters

Create src/evaluation/metrics.ts.

export type RunRecord = {
  scenariosGenerated: number;
  scenariosAccepted: number;
  testsGenerated: number;
  testsAccepted: number;
  diagnoses: Array<{ ai: string; human: string }>;
  patchesApproved: number;
  patchesResolved: number;
  costUsd: number;
  reviewMinutes: number;
};

const percent = (part: number, whole: number) =>
  whole === 0 ? 0 : Math.round((part / whole) * 100);

export function calculateMetrics(runs: RunRecord[]) {
  const total = (pick: (run: RunRecord) => number) =>
    runs.reduce((sum, run) => sum + pick(run), 0);
  const diagnoses = runs.flatMap((run) => run.diagnoses);

  return {
    scenarioValidityRate: percent(total((r) => r.scenariosAccepted), total((r) => r.scenariosGenerated)),
    testAcceptanceRate: percent(total((r) => r.testsAccepted), total((r) => r.testsGenerated)),
    classificationAccuracy: percent(
      diagnoses.filter((item) => item.ai === item.human).length,
      diagnoses.length,
    ),
    healingSuccessRate: percent(total((r) => r.patchesResolved), total((r) => r.patchesApproved)),
    averageCostUsd: runs.length === 0 ? 0 : total((r) => r.costUsd) / runs.length,
    averageReviewMinutes: runs.length === 0 ? 0 : total((r) => r.reviewMinutes) / runs.length,
  };
}

Metric	Question it answers
Scenario validity rate	How many scenarios survive review?
Test acceptance rate	How many tests need little or no editing?
Classification accuracy	Does the AI’s diagnosis match a person’s?
Healing success rate	Do approved fixes really solve the problem?
Cost and review time	Is it worth it?

Add execution reliability too, by repeating controlled runs and comparing the results. You learned about this measuring approach in LLM Evaluation Metrics for Testers.

---

🛠️ The Rollout Order

1. Run lint, type checks and unit tests on every pull request.
2. Add a small and stable Playwright smoke suite.
3. Publish HTML reports and failure artifacts.
4. Keep model generation in a separate, approved job.
5. Pin dependencies, and store secrets in the CI secret store.
6. Add cost, latency, failure and approval metrics before scaling.

---

🌍 Real-Life Example: The Flaky Gate

A team runs model-based test generation on every pull request. When the provider has a slow afternoon, nobody can merge anything. They move the model job to a nightly schedule, and the pull request checks return to being fast and predictable.

---

🧪 Try It Yourself

1. Add the workflow file and commit it.
2. Push to a private GitHub repository.
3. Watch the job and open the uploaded artifacts.
4. Break a unit test on a branch, and confirm that CI fails.
5. Fill in the metrics for three runs of your agent.

---

📋 Ready-to-Copy Prompt

You are a senior DevOps engineer reviewing a CI workflow for a QA AI agent.

Workflow:
[paste the YAML with no secrets]

Check permissions, secret exposure on pull requests, caching, timeouts, and
whether model calls could block a merge.

Return a table with: Risk, Where It Is and Smallest Fix.
Remind me to pin third-party actions according to my team's policy.

---

⚠️ Common Mistakes

• Making merges depend on a live model
• Giving CI jobs write access that they do not need
• Exposing real keys to pull request code
• Throwing away test evidence after failures
• Reporting only how many tests were generated

---

🎯 Key Takeaways

• The main pipeline uses mocks and stable tests.
• Run model jobs separately, with approval.
• Give jobs the least access they need.
• Measure quality as well as quantity.

---

🎤 Interview Questions

1. How would you add an AI test agent to CI?

Model answer: I would run lint, type checks and mock-based tests on each pull request, with a small smoke suite. Model generation would run in a separate approved job.

2. Why keep LLM calls out of the critical path?

Model answer: They add cost, delay and variation, and can fail for outside reasons. A merge should depend on predictable checks.

3. What metrics would you track for a QA agent?

Model answer: I would track scenario validity, test acceptance, classification accuracy, healing success, reliability, cost, latency and review time.

4. How do you protect secrets in CI?

Model answer: I use the platform’s secret store, limit job permissions and avoid exposing secrets to untrusted pull request code. Tests use synthetic data.

5. What does a high test acceptance rate tell you?

Model answer: Reviewers accept most generated tests with few edits. It should be read together with correctness, because weak tests could also be accepted.

---

📝 Assignment

Add a nightly workflow that runs only by schedule or manual trigger. Have it do nothing more than print a message for now. Write down which secrets it would need and who should be able to run it.

---

❓ FAQ

Why is the smoke test only one test?

A small suite gives fast feedback. Add stable tests as your application grows.

Do I have to use GitHub Actions?

No. The same steps work in other CI systems.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Evaluation: Security, Reliability and a Test Pack.