Measure AI Testing ROI: Jira and GitHub Integration for QA Agents

An agent earns its place when its work reaches the tools your team already uses, and when you can prove that it helps. In this final lesson of the level you will file bugs in Jira, open pull requests on GitHub and measure AI testing ROI. An honest calculation includes the costs that are easy to forget.

---

💡 Connect the Agent to Your Tools

Output	Destination	Rule
Confirmed product defect	Jira bug	A person confirms it first
Generated test	GitHub pull request	A person reviews and merges it
Run report	Linked from both	Evidence stays available

---

🔍 File Bugs Without Duplicates

Create src/integrations/jira.ts.

import { createHash } from 'node:crypto';
import { redact } from '../core/guardrails.js';
import type { FailedTest } from '../core/types.js';
import type { FailureAnalysis } from '../schemas/failure.schema.js';
import type { ServiceStore } from '../service/store.js';

export type JiraConfig = { baseUrl: string; email: string; apiToken: string; projectKey: string };

export function buildBugPayload(
  projectKey: string,
  runId: string,
  failure: FailedTest,
  analysis: FailureAnalysis,
) {
  const text = redact(
    [`Run: ${runId}`, `Diagnosis (${analysis.confidence}): ${analysis.summary}`, failure.error].join('\n\n'),
  );
  return {
    fields: {
      project: { key: projectKey },
      issuetype: { name: 'Bug' },
      summary: `[QA Agent] ${failure.title}`.slice(0, 250),
      labels: ['qa-agent'],
      description: {
        type: 'doc',
        version: 1,
        content: [{ type: 'paragraph', content: [{ type: 'text', text }] }],
      },
    },
  };
}

export class JiraClient {
  constructor(
    private readonly config: JiraConfig,
    private readonly store: ServiceStore,
  ) {}

  // Only a confirmed product defect becomes a bug, and each one is filed once.
  async createBugOnce(
    runId: string,
    failure: FailedTest,
    analysis: FailureAnalysis,
    approvedBy: string,
  ): Promise<string> {
    if (analysis.category !== 'product_defect') {
      throw new Error('Only product defects can be filed as bugs');
    }
    const fingerprint = createHash('sha256').update(failure.title + failure.error).digest('hex');
    const settingKey = `jira:${fingerprint}`;
    const existing = this.store.getSetting(settingKey);
    if (existing) {
      return existing;
    }

    const credentials = Buffer.from(`${this.config.email}:${this.config.apiToken}`).toString('base64');
    const response = await fetch(`${this.config.baseUrl}/rest/api/3/issue`, {
      method: 'POST',
      headers: {
        Authorization: `Basic ${credentials}`,
        Accept: 'application/json',
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(buildBugPayload(this.config.projectKey, runId, failure, analysis)),
    });
    if (!response.ok) {
      throw new Error(`Jira returned ${response.status}`);
    }
    const { key } = (await response.json()) as { key: string };
    this.store.setSetting(settingKey, key);
    this.store.addAudit(approvedBy, 'jira.bug_created', runId, key);
    return key;
  }
}

Safeguard	Why
Only product_defect	Test problems are not product bugs
approvedBy is required	A person has confirmed it
Fingerprint	One failure creates one bug
redact	Secrets stay out of tickets

You practised writing bug reports in Write Better Bug Reports with AI. Use a Jira API token for an account that can create issues only in the test project.

---

🧩 Deliver Tests as Pull Requests

Create src/integrations/github.ts.

import { isSafeGeneratedPath } from '../core/guardrails.js';
import type { GeneratedFile } from '../schemas/generation.schema.js';

export type GitHubConfig = {
  apiUrl: string;
  owner: string;
  repo: string;
  token: string;
  baseBranch: string;
};

export class GitHubClient {
  constructor(private readonly config: GitHubConfig) {}

  private async call<T>(method: string, path: string, body?: unknown): Promise<T> {
    const { apiUrl, owner, repo, token } = this.config;
    const response = await fetch(`${apiUrl}/repos/${owner}/${repo}${path}`, {
      method,
      headers: {
        Authorization: `Bearer ${token}`,
        Accept: 'application/vnd.github+json',
        'Content-Type': 'application/json',
      },
      body: body === undefined ? undefined : JSON.stringify(body),
    });
    if (!response.ok) {
      throw new Error(`GitHub ${method} ${path} returned ${response.status}`);
    }
    return (await response.json()) as T;
  }

  // Generated tests reach the main branch only through a reviewed pull request.
  async openPullRequest(runId: string, file: GeneratedFile): Promise<string> {
    if (!isSafeGeneratedPath(file.path)) {
      throw new Error(`Path is not approved: ${file.path}`);
    }
    const { baseBranch } = this.config;
    const branch = `qa-agent/${runId}`;

    const base = await this.call<{ object: { sha: string } }>(
      'GET',
      `/git/ref/heads/${baseBranch}`,
    );
    await this.call('POST', '/git/refs', { ref: `refs/heads/${branch}`, sha: base.object.sha });
    await this.call('PUT', `/contents/${file.path}`, {
      message: `Add generated tests from ${runId}`,
      content: Buffer.from(file.content).toString('base64'),
      branch,
    });
    const pull = await this.call<{ html_url: string }>('POST', '/pulls', {
      title: `QA Agent: generated tests from ${runId}`,
      head: branch,
      base: baseBranch,
      body: 'Generated by the QA agent. Please review the assertions before merging.',
      draft: true,
    });
    return pull.html_url;
  }
}

The pull request is a draft, so it cannot be merged by accident. Use a fine-grained token limited to this one repository, and turn on branch protection for your main branch.

---

🛠️ Test Without Touching Real Systems

Create src/evaluation/roi.ts, which the next section uses. Then create tests/unit/integrations.test.ts. It runs a small local stand-in, so your tests never create a real ticket.

export type RoiInput = {
  runsPerMonth: number;
  minutesWithoutAgent: number;
  reviewMinutesWithAgent: number;
  hourlyCostUsd: number;
  modelCostPerRunUsd: number;
  platformCostPerMonthUsd: number;
  maintenanceHoursPerMonth: number;
};

export function calculateRoi(input: RoiInput) {
  const minutesSaved = input.minutesWithoutAgent - input.reviewMinutesWithAgent;
  const hoursSaved = (input.runsPerMonth * minutesSaved) / 60;
  const benefitUsd = hoursSaved * input.hourlyCostUsd;
  const costUsd =
    input.runsPerMonth * input.modelCostPerRunUsd +
    input.platformCostPerMonthUsd +
    input.maintenanceHoursPerMonth * input.hourlyCostUsd;
  return {
    hoursSaved,
    benefitUsd,
    costUsd,
    netUsd: benefitUsd - costUsd,
    roiPercent: costUsd === 0 ? 0 : Math.round(((benefitUsd - costUsd) / costUsd) * 100),
  };
}

import assert from 'node:assert/strict';
import { createServer, type Server } from 'node:http';
import type { AddressInfo } from 'node:net';
import { after, before, test } from 'node:test';
import { calculateRoi } from '../../src/evaluation/roi.js';
import { GitHubClient } from '../../src/integrations/github.js';
import { buildBugPayload, JiraClient } from '../../src/integrations/jira.js';
import type { FailureAnalysis } from '../../src/schemas/failure.schema.js';
import { ServiceStore } from '../../src/service/store.js';

// A local stand-in, so these tests never touch a real Jira or GitHub.
const calls: string[] = [];
let server: Server;
let url: string;

before(async () => {
  server = createServer((request, response) => {
    calls.push(`${request.method} ${request.url}`);
    response.setHeader('Content-Type', 'application/json');
    response.end(
      JSON.stringify({ key: 'QA-1', object: { sha: 'abc' }, html_url: 'https://example.test/pull/1' }),
    );
  });
  await new Promise<void>((done) => {
    server.listen(0, '127.0.0.1', () => done());
  });
  url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
after(() => server.close());

const failure = {
  title: 'login fails',
  error: 'password=hunter2 rejected',
  tracePaths: [],
  screenshotPaths: [],
};
const analysis: FailureAnalysis = {
  category: 'product_defect',
  summary: 'The login button is broken.',
  evidence: [],
  confidence: 'high',
  alternatives: [],
};
const jiraConfig = () => ({ baseUrl: url, email: 'a@example.test', apiToken: 't', projectKey: 'QA' });

test('a bug report never contains secrets', () => {
  const payload = JSON.stringify(buildBugPayload('QA', 'run-1', failure, analysis));
  assert.doesNotMatch(payload, /hunter2/);
});

test('the same failure is filed only once', async () => {
  calls.length = 0;
  const jira = new JiraClient(jiraConfig(), new ServiceStore(':memory:'));
  assert.equal(await jira.createBugOnce('run-1', failure, analysis, 'ben'), 'QA-1');
  assert.equal(await jira.createBugOnce('run-2', failure, analysis, 'ben'), 'QA-1');
  assert.equal(calls.length, 1);
});

test('test defects are never filed as product bugs', async () => {
  const jira = new JiraClient(jiraConfig(), new ServiceStore(':memory:'));
  await assert.rejects(
    jira.createBugOnce('run-1', failure, { ...analysis, category: 'test_defect' }, 'ben'),
    /Only product defects/,
  );
});

test('generated tests arrive as a pull request, in the right order', async () => {
  calls.length = 0;
  const github = new GitHubClient({ apiUrl: url, owner: 'o', repo: 'r', token: 't', baseBranch: 'main' });
  await github.openPullRequest('run-1', { path: 'tests/generated/login.spec.ts', content: 'x' });
  assert.deepEqual(calls, [
    'GET /repos/o/r/git/ref/heads/main',
    'POST /repos/o/r/git/refs',
    'PUT /repos/o/r/contents/tests/generated/login.spec.ts',
    'POST /repos/o/r/pulls',
  ]);
});

test('calculates return on investment', () => {
  const result = calculateRoi({
    runsPerMonth: 40,
    minutesWithoutAgent: 60,
    reviewMinutesWithAgent: 15,
    hourlyCostUsd: 30,
    modelCostPerRunUsd: 0.5,
    platformCostPerMonthUsd: 50,
    maintenanceHoursPerMonth: 4,
  });
  assert.equal(result.hoursSaved, 30);
  assert.equal(result.benefitUsd, 900);
  assert.equal(result.costUsd, 190);
  assert.equal(result.netUsd, 710);
});

After these pass, try the integrations against a sandbox Jira project and a practice repository, and read each provider’s current API documentation first.

---

📌 Measure AI Testing ROI Honestly

ROI compares what you gain with what you spend. This calculation uses example figures.

Input	Example
Runs per month	40
Minutes per task without the agent	60
Review minutes with the agent	15
Hourly cost	$30
Model cost per run	$0.50
Platform cost per month	$50
Maintenance hours per month	4

Result	Value
Hours saved	30
Benefit	$900
Cost	$190
Net	$710

These numbers are only an illustration. Use your own measurements, and find them by timing the same tasks with and without the agent.

---

🔑 Measure More Than Money

Measure	What it tells you
Test acceptance rate	Whether the output is usable
Defects found by agent tests	Whether the tests catch real problems
False bugs filed	Whether people’s time is being wasted
Coverage of agreed requirements	Whether it is closing gaps
Tester feedback	Whether people want to keep using it

Time saved is only valuable if quality stays the same or improves. Your metrics from the AI agent CI pipeline feed directly into this.

---

🌍 Real-Life Example: The Number That Fell Apart

A team claims that its agent saves 200 hours a month. Someone asks about review time, and it turns out the generated tests needed almost as long to fix as they would have taken to write. A proper measurement shows a smaller but real saving, and it is one the team can defend.

---

🧪 Try It Yourself

1. Create the files, and run the unit tests.
2. Time one task with and without your agent.
3. Fill in the ROI calculation with your numbers.
4. List every cost that you might have forgotten.
5. Write one honest sentence about the value.

---

📋 Ready-to-Copy Prompt

You are a sceptical finance partner reviewing the ROI of a QA AI agent.

Measurements:
[paste your real numbers and how you measured them]

Find any cost I left out, any benefit I overstated and any figure that
was estimated instead of measured. Do not change or invent numbers.

Then write a short, honest summary for a director.

---

⚠️ Common Mistakes

• Filing every failure as a product bug
• Giving integration tokens more power than needed
• Letting the agent merge its own code
• Counting time saved but not review time
• Reporting estimates as if they were measurements

---

🎯 Key Takeaways

• Deliver results into the tools people already use.
• Prevent duplicate tickets with a fingerprint.
• Generated code arrives as a reviewable pull request.
• Honest ROI includes every real cost.

---

🎤 Interview Questions

1. How would you integrate an AI agent with Jira?

Model answer: I would file only human-confirmed product defects, with redacted evidence and a fingerprint to prevent duplicates, using a narrowly scoped token.

2. How should an agent deliver generated tests?

Model answer: It should open a draft pull request. People review and merge it, and branch protection means the agent cannot merge its own work.

3. How do you measure ROI for an AI testing tool?

Model answer: I compare the measured time for the same tasks, then subtract model, platform, review and maintenance costs. I also track quality.

4. What costs are often forgotten?

Model answer: Human review, fixing poor output, maintenance, incidents and training. They can remove much of the apparent saving.

5. How do you avoid duplicate bugs?

Model answer: I create a fingerprint from stable failure details, record it when a bug is filed and return the existing ticket if the same failure comes back.

---

📝 Assignment

Connect your agent to a sandbox Jira project and a practice repository. Measure five real tasks with and without it. Write a one-page summary of the value, including at least one thing that did not work.

---

❓ FAQ

Should the agent close Jira tickets too?

Not at first. Creating a ticket is easy to undo. Closing work needs a much higher level of trust.

What if the ROI is negative?

That is a useful result. Narrow the agent to the tasks where it helps, or stop using it.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with QA Career Path with AI: From Manual Tester to QA Architect.