AI Agent as a Service: Wrap Your QA Agent in an API

Until now, a person has had to type yes in a terminal. An AI agent as a service works differently. A team member sends a request to an API, a background worker runs the job and an approver decides from wherever they are. You will reuse your orchestrator almost unchanged, because you kept it independent of how it is started.

---

💡 The API for an AI Agent as a Service

Request	Purpose	Who may use it
POST /v1/runs	Start a run	Requester
GET /v1/runs/:id	Check its status	Viewer
GET /v1/runs/:id/report	Read the report	Viewer
GET /v1/approvals	See pending approvals	Approver
POST /v1/approvals/:id	Approve or reject	Approver
GET /healthz and /readyz	Check the service	Anyone

Notice that a request names an environment, and it never gives a URL. People choose from approved environments, so nobody can point the agent at an arbitrary site.

Install the web framework.

npm install express
npm install -D @types/express

Create src/service/config.ts.

import 'dotenv/config';
import * as z from 'zod';

const ServiceSchema = z.object({
  SERVICE_HOST: z.string().default('127.0.0.1'),
  SERVICE_PORT: z.coerce.number().int().positive().default(8080),
  DATABASE_PATH: z.string().default('data/agent.db'),
  APPROVAL_TIMEOUT_MINUTES: z.coerce.number().positive().default(60),
  AUTH_MODE: z.enum(['local-dev', 'token']).default('token'),
  MAX_LLM_CALLS_PER_RUN: z.coerce.number().int().positive().default(20),
  MAX_TOKENS_PER_RUN: z.coerce.number().int().positive().default(200_000),
  MAX_DAILY_COST_USD: z.coerce.number().positive().default(10),
  INPUT_COST_PER_MILLION_USD: z.coerce.number().nonnegative().default(0),
  OUTPUT_COST_PER_MILLION_USD: z.coerce.number().nonnegative().default(0),
  RUNS_PER_HOUR_PER_USER: z.coerce.number().int().positive().default(10),
  SLACK_SIGNING_SECRET: z.string().default(''),
  SLACK_WEBHOOK_URL: z.string().default(''),
  SLACK_APPROVERS: z.string().default('{}'),
});

export type ServiceConfig = z.infer<typeof ServiceSchema>;

export function loadServiceConfig(): ServiceConfig {
  return ServiceSchema.parse(process.env);
}

Create src/service/security.ts. Roles are defined here, and Lesson 83 adds real sign-in.

import type { Request } from 'express';

export type Role = 'viewer' | 'requester' | 'approver' | 'admin';
export type Permission = 'runs:read' | 'runs:create' | 'approvals:decide' | 'ops:manage';
export type Actor = { id: string; role: Role };

const ROLE_PERMISSIONS: Record<Role, Permission[]> = {
  viewer: ['runs:read'],
  requester: ['runs:read', 'runs:create'],
  approver: ['runs:read', 'approvals:decide'],
  admin: ['runs:read', 'runs:create', 'approvals:decide', 'ops:manage'],
};

export function isAllowed(actor: Actor, permission: Permission): boolean {
  return ROLE_PERMISSIONS[actor.role].includes(permission);
}

export interface Security {
  authenticate(request: Request): Actor | null;
}

// For your own machine only. It trusts a header that anyone could set.
export class LocalDevSecurity implements Security {
  authenticate(request: Request): Actor | null {
    const id = request.header('x-dev-user');
    return id ? { id, role: 'admin' } : null;
  }
}

---

🔍 Approvals without a Terminal

Your Approver interface from Validate AI-Generated Code pays off here. Create src/service/stored-approver.ts.

import { setTimeout as sleep } from 'node:timers/promises';
import type { ApprovalRequest, Approver } from '../core/approval.js';
import type { ApprovalRow, ServiceStore } from './store.js';

export type ApprovalNotifier = (approval: ApprovalRow) => Promise<void>;

export class StoredApprover implements Approver {
  constructor(
    private readonly store: ServiceStore,
    private readonly runId: string,
    private readonly timeoutMs: number,
    private readonly notify?: ApprovalNotifier,
    private readonly pollMs = 1000,
  ) {}

  async decide({ action, summary }: ApprovalRequest): Promise<boolean> {
    const expiresAt = Date.now() + this.timeoutMs;
    const approval = this.store.createApproval(this.runId, action, summary, expiresAt);
    await this.notify?.(approval).catch((error: unknown) => {
      this.store.addAudit('system', 'notification.failed', this.runId, String(error));
    });

    while (Date.now() < expiresAt) {
      const current = this.store.getApproval(approval.id);
      if (current && current.status !== 'pending') {
        return current.status === 'approved';
      }
      await sleep(this.pollMs);
    }
    // Nobody answered in time, so the action does not happen.
    this.store.decideApproval(approval.id, 'expired', 'system');
    return false;
  }

  close(): void {}
}

export type DecisionResult = 'ok' | 'not_found' | 'already_decided' | 'self_approval';

export function recordDecision(
  store: ServiceStore,
  approvalId: string,
  actorId: string,
  approved: boolean,
): DecisionResult {
  const approval = store.getApproval(approvalId);
  const run = approval ? store.getRun(approval.run_id) : undefined;
  if (!approval || !run) {
    return 'not_found';
  }
  if (run.requested_by === actorId) {
    return 'self_approval';
  }
  if (!store.decideApproval(approvalId, approved ? 'approved' : 'rejected', actorId)) {
    return 'already_decided';
  }
  store.addAudit(actorId, approved ? 'approval.approved' : 'approval.rejected', run.id, approval.action);
  return 'ok';
}

Two rules matter here. An unanswered request counts as a refusal, and a person cannot approve their own run.

---

🧩 Run the Workflow in a Worker

Create src/service/run-workflow.ts. It connects the agents from AI Agent Orchestrator.

import { ExplorerAgent } from '../agents/explorer.agent.js';
import { FailureAnalystAgent } from '../agents/failure-analyst.agent.js';
import { GeneratorAgent } from '../agents/generator.agent.js';
import { HealerAgent } from '../agents/healer.agent.js';
import { PlannerAgent } from '../agents/planner.agent.js';
import { ReporterAgent } from '../agents/reporter.agent.js';
import { RequirementParserAgent } from '../agents/requirement-parser.agent.js';
import type { Env } from '../config/env.js';
import { buildProjectContext } from '../config/project.js';
import type { Approver } from '../core/approval.js';
import { Orchestrator } from '../core/orchestrator.js';
import { createState } from '../core/workflow-state.js';
import type { LlmClient } from '../llm/llm-client.js';
import { createLlmClient } from '../llm/provider-adapter.js';
import { PlaywrightBrowserTool } from '../tools/browser.tool.js';
import { FileSystemTool } from '../tools/filesystem.tool.js';
import { TestRunnerTool } from '../tools/test-runner.tool.js';
import type { RunRow } from './store.js';

export type WorkflowOutcome = {
  status: 'completed' | 'stopped';
  stopReason?: string;
  reportPath: string;
  stateJson: string;
};

export type WorkflowRunner = (run: RunRow, approver: Approver) => Promise<WorkflowOutcome>;

export function createWorkflowRunner(
  root: string,
  env: Env,
  wrapLlm: (client: LlmClient, run: RunRow) => LlmClient = (client) => client,
): WorkflowRunner {
  const context = buildProjectContext(env);

  return async (run, approver) => {
    const llm = wrapLlm(createLlmClient(env), run);
    const orchestrator = new Orchestrator(
      root,
      context,
      {
        parser: new RequirementParserAgent(llm),
        explorer: new ExplorerAgent(new PlaywrightBrowserTool(true), context.allowedDomains),
        planner: new PlannerAgent(llm),
        generator: new GeneratorAgent(llm),
        analyst: new FailureAnalystAgent(llm),
        healer: new HealerAgent(llm),
        reporter: new ReporterAgent(root),
      },
      {
        files: new FileSystemTool(root),
        runner: new TestRunnerTool(root, env.ALLOW_TEST_EXECUTION === 'true'),
        approver,
      },
    );
    const state = createState(run.id, run.requirement, llm.providerName);
    const reportPath = await orchestrator.run(state);
    return {
      status: state.status === 'completed' ? 'completed' : 'stopped',
      stopReason: state.stopReason,
      reportPath,
      stateJson: JSON.stringify(state),
    };
  };
}

Create src/service/worker.ts.

import { setTimeout as sleep } from 'node:timers/promises';
import { StoredApprover, type ApprovalNotifier } from './stored-approver.js';
import type { WorkflowRunner } from './run-workflow.js';
import type { RunRow, ServiceStore } from './store.js';

export type WorkerOptions = {
  approvalTimeoutMs: number;
  notify?: ApprovalNotifier;
  onFinished?: (run: RunRow, status: string) => void | Promise<void>;
  leaseMs?: number;
  pollMs?: number;
};

export class Worker {
  private running = false;
  private readonly leaseMs: number;

  constructor(
    private readonly store: ServiceStore,
    private readonly runWorkflow: WorkflowRunner,
    private readonly options: WorkerOptions,
  ) {
    this.leaseMs = options.leaseMs ?? 60_000;
  }

  // Runs one queued job. Returns false when there was nothing to do.
  async tick(): Promise<boolean> {
    if (this.store.isPaused()) {
      return false;
    }
    const run = this.store.claimNextRun(this.leaseMs);
    if (!run) {
      return false;
    }

    const heartbeat = setInterval(
      () => this.store.renewLease(run.id, this.leaseMs),
      this.leaseMs / 3,
    );
    try {
      this.store.addAudit('worker', 'run.started', run.id);
      const approver = new StoredApprover(
        this.store,
        run.id,
        this.options.approvalTimeoutMs,
        this.options.notify,
        this.options.pollMs,
      );
      const outcome = await this.runWorkflow(run, approver);
      this.store.finishRun(run.id, outcome);
      this.store.addAudit('worker', `run.${outcome.status}`, run.id, outcome.stopReason);
      await this.options.onFinished?.(run, outcome.status);
    } catch (error) {
      const stopReason = error instanceof Error ? error.message : String(error);
      this.store.finishRun(run.id, { status: 'stopped', stopReason });
      this.store.addAudit('worker', 'run.failed', run.id, stopReason);
      await this.options.onFinished?.(run, 'stopped');
    } finally {
      clearInterval(heartbeat);
    }
    return true;
  }

  // One run at a time: runs share a generated tests folder and a results file.
  async start(): Promise<void> {
    this.running = true;
    this.store.interruptExpiredRuns();
    while (this.running) {
      const worked = await this.tick();
      if (!worked) {
        await sleep(this.options.pollMs ?? 1000);
      }
    }
  }

  stop(): void {
    this.running = false;
  }
}

The heartbeat renews the lease while a run is healthy. If the process dies, the lease expires and the run is marked as interrupted.

---

🛠️ Build the API

Create src/service/api.ts.

import { readFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import express, {
  type Express,
  type NextFunction,
  type Request,
  type RequestHandler,
  type Response,
} from 'express';
import * as z from 'zod';
import { isAllowed, type Actor, type Permission, type Security } from './security.js';
import { recordDecision } from './stored-approver.js';
import type { ServiceStore } from './store.js';

const CreateRunSchema = z.object({
  requirement: z.string().min(1).max(20_000),
  environment: z.string().min(1),
});
const DecisionSchema = z.object({ decision: z.enum(['approve', 'reject']) });

export type Guard = (permission: Permission) => RequestHandler;

export type ApiOptions = {
  root: string;
  store: ServiceStore;
  security: Security;
  environments: string[];
  allowRun?: (actorId: string) => boolean;
  mount?: (app: Express, guard: Guard) => void;
};

const actorOf = (response: Response) => response.locals.actor as Actor;

export function createApi(options: ApiOptions): Express {
  const { root, store, security, environments } = options;
  const app = express();
  app.disable('x-powered-by');

  const guard: Guard = (permission) => (request, response, next) => {
    const actor = security.authenticate(request);
    if (!actor) {
      response.status(401).json({ error: 'Authentication required' });
    } else if (!isAllowed(actor, permission)) {
      response.status(403).json({ error: 'Not allowed' });
    } else {
      response.locals.actor = actor;
      next();
    }
  };

  options.mount?.(app, guard);
  app.use(express.json({ limit: '64kb' }));

  app.get('/healthz', (_request, response) => {
    response.json({ status: 'ok' });
  });

  app.get('/readyz', (_request, response) => {
    response.json({ status: store.ping() ? 'ready' : 'not_ready', paused: store.isPaused() });
  });

  app.post('/v1/runs', guard('runs:create'), (request, response) => {
    const body = CreateRunSchema.safeParse(request.body);
    if (!body.success) {
      response.status(400).json({ error: 'Invalid request body' });
      return;
    }
    if (!environments.includes(body.data.environment)) {
      response.status(400).json({ error: 'Environment is not approved' });
      return;
    }
    const actor = actorOf(response);
    if (options.allowRun && !options.allowRun(actor.id)) {
      response.status(429).json({ error: 'Too many runs. Try again later.' });
      return;
    }
    const run = store.createRun({
      requirement: body.data.requirement,
      environment: body.data.environment,
      requestedBy: actor.id,
      idempotencyKey: request.header('idempotency-key'),
    });
    store.addAudit(actor.id, 'run.requested', run.id);
    response.status(202).json({ id: run.id, status: run.status });
  });

  app.get('/v1/runs/:id', guard('runs:read'), (request, response) => {
    const run = store.getRun(String(request.params.id));
    if (!run) {
      response.status(404).json({ error: 'Run not found' });
      return;
    }
    response.json({
      id: run.id,
      status: run.status,
      requestedBy: run.requested_by,
      stopReason: run.stop_reason,
      reportPath: run.report_path,
    });
  });

  app.get('/v1/runs/:id/report', guard('runs:read'), async (request, response) => {
    const run = store.getRun(String(request.params.id));
    if (!run?.report_path) {
      response.status(404).json({ error: 'Report not found' });
      return;
    }
    response.type('text/markdown').send(await readFile(resolve(root, run.report_path), 'utf8'));
  });

  app.get('/v1/approvals', guard('approvals:decide'), (_request, response) => {
    response.json(store.listPendingApprovals());
  });

  app.post('/v1/approvals/:id', guard('approvals:decide'), (request, response) => {
    const body = DecisionSchema.safeParse(request.body);
    if (!body.success) {
      response.status(400).json({ error: 'Decision must be approve or reject' });
      return;
    }
    const result = recordDecision(
      store,
      String(request.params.id),
      actorOf(response).id,
      body.data.decision === 'approve',
    );
    const statusCodes = { ok: 200, not_found: 404, already_decided: 409, self_approval: 403 };
    response.status(statusCodes[result]).json({ result });
  });

  app.use((error: unknown, _request: Request, response: Response, next: NextFunction) => {
    if (response.headersSent) {
      next(error);
      return;
    }
    const badInput = error instanceof SyntaxError || (error as { status?: number }).status === 413;
    response.status(badInput ? 400 : 500).json({ error: badInput ? 'Invalid request' : 'Server error' });
  });

  return app;
}

---

📌 Start the Service

Create src/service/main.ts.

import { loadEnv } from '../config/env.js';
import { createApi } from './api.js';
import { loadServiceConfig } from './config.js';
import { createWorkflowRunner } from './run-workflow.js';
import { LocalDevSecurity } from './security.js';
import { ServiceStore } from './store.js';
import { Worker } from './worker.js';

// Temporary version. Lesson 88 adds real security, limits and monitoring.
const root = process.cwd();
const env = loadEnv();
const config = loadServiceConfig();
if (config.SERVICE_HOST !== '127.0.0.1') {
  throw new Error('This version may run only on 127.0.0.1');
}

const store = new ServiceStore(config.DATABASE_PATH);
const worker = new Worker(store, createWorkflowRunner(root, env), {
  approvalTimeoutMs: config.APPROVAL_TIMEOUT_MINUTES * 60_000,
});
const app = createApi({ root, store, security: new LocalDevSecurity(), environments: ['demo'] });

app.listen(config.SERVICE_PORT, config.SERVICE_HOST, () => {
  console.log(`Service running on port ${config.SERVICE_PORT}`);
});
void worker.start();

Add the script, then start the demo and the service in separate terminals.

npm pkg set scripts.service="tsx src/service/main.ts"
npm run demo
npm run service

In a third terminal, request a run as Asha.

curl -X POST http://127.0.0.1:8080/v1/runs -H "Content-Type: application/json" -H "x-dev-user: asha" -d "{\"requirement\":\"A user can sign in.\",\"environment\":\"demo\"}"

List what is waiting, acting as Ben, and then approve each one. Do this twice, because there are two approvals.

curl http://127.0.0.1:8080/v1/approvals -H "x-dev-user: ben"
curl -X POST http://127.0.0.1:8080/v1/approvals/YOUR-APPROVAL-ID -H "Content-Type: application/json" -H "x-dev-user: ben" -d "{\"decision\":\"approve\"}"

Finally, check the run with GET /v1/runs/YOUR-RUN-ID. Try approving as asha, and you should get a 403 response.

---

🔑 Test the Worker

Create tests/unit/worker.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import type { WorkflowRunner } from '../../src/service/run-workflow.js';
import { ServiceStore } from '../../src/service/store.js';
import { recordDecision, StoredApprover } from '../../src/service/stored-approver.js';
import { Worker } from '../../src/service/worker.js';

const queue = (store: ServiceStore) =>
  store.createRun({ requirement: 'Login', environment: 'demo', requestedBy: 'asha' });

const finished: WorkflowRunner = async () => ({
  status: 'completed',
  reportPath: 'reports/x.md',
  stateJson: '{}',
});

test('the worker completes a queued run', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  await new Worker(store, finished, { approvalTimeoutMs: 1000 }).tick();
  assert.equal(store.getRun(run.id)?.status, 'completed');
});

test('a crash is recorded as a stopped run', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  const crash: WorkflowRunner = async () => {
    throw new Error('Browser crashed');
  };
  await new Worker(store, crash, { approvalTimeoutMs: 1000 }).tick();
  assert.equal(store.getRun(run.id)?.status, 'stopped');
  assert.equal(store.getRun(run.id)?.stop_reason, 'Browser crashed');
});

test('a paused service does not start new runs', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  store.setSetting('paused', 'true');
  await new Worker(store, finished, { approvalTimeoutMs: 1000 }).tick();
  assert.equal(store.getRun(run.id)?.status, 'queued');
});

test('an unanswered approval counts as a no', async () => {
  const store = new ServiceStore(':memory:');
  const approver = new StoredApprover(store, queue(store).id, 20, undefined, 5);
  assert.equal(await approver.decide({ action: 'run-generated-tests', summary: 'x' }), false);
});

test('a person cannot approve their own run', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  const approval = store.createApproval(run.id, 'x', 'y', Date.now() + 1000);
  assert.equal(recordDecision(store, approval.id, 'asha', true), 'self_approval');
  assert.equal(recordDecision(store, approval.id, 'ben', true), 'ok');
  assert.equal(recordDecision(store, 'apr-missing', 'ben', true), 'not_found');
});

---

🌍 Real-Life Example: Approve at Lunch

A QA lead is away from their desk when a run reaches its approval step. With a terminal prompt, the run would simply wait for one person at one machine. With stored approvals, a colleague can review it from another machine, and every decision is recorded.

---

🧪 Try It Yourself

1. Create the files and run the unit tests.
2. Run the service and complete one run.
3. Try to approve your own run.
4. Send an environment called production.
5. Send broken JSON and read the response.

---

📋 Ready-to-Copy Prompt

You are a senior API engineer reviewing a service that wraps a QA agent.

Code:
[paste your routes with no secrets]

Check input validation, error messages, status codes and anything that lets
a caller reach systems outside the approved list.

Return a table with: Route, Problem and Smallest Fix.
Do not suggest a new framework.

---

⚠️ Common Mistakes

• Letting callers provide a target URL
• Treating a missing approval as a yes
• Returning stack traces to callers
• Allowing people to approve their own work
• Exposing local-only security on a network

---

🎯 Key Takeaways

• The API accepts work, and the worker runs it.
• Approvals become stored records that wait.
• Unanswered approvals count as a no.
• The orchestrator did not need to change.

---

🎤 Interview Questions

1. How do you turn a CLI agent into a service?

Model answer: I add an API to accept requests, a queue and worker for the slow work and a database for state. I keep the core workflow independent of how it starts.

2. How do approvals work in a service?

Model answer: The workflow creates an approval record and waits. A person decides through the API, and the workflow continues. If it times out, the answer is no.

3. Why not let users pass a target URL?

Model answer: It could point the agent at unapproved or internal systems. I accept the name of a configured environment and map it to a known target.

4. What is the difference between health and readiness?

Model answer: Health shows that the process is alive. Readiness shows that it can do work, such as reaching its database and not being paused.

5. What is separation of duties?

Model answer: The person requesting an action cannot approve it. This reduces the risk of mistakes and misuse.

---

📝 Assignment

Add a GET /v1/runs route that returns a requester’s last ten runs. Protect it with runs:read, validate its query and add a unit test.

---

❓ FAQ

Why does the worker run in the same process as the API?

It keeps the first release simple. The two talk only through the database, so they can be separated later.

How do I send requests from Windows?

The commands work in Windows Terminal with curl.exe. A tool such as Postman also works well.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Agent Workflow State: Save Runs, Traces and Evidence.