LLM Cost Control: Token Budgets and Rate Limits for AI Agents

A bug in an agent loop can spend a surprising amount of money before anyone notices. LLM cost control sets limits that your code enforces, so a mistake stops early and cheaply. In this lesson you will wrap the model client with budgets, add a daily spending cap and limit how often each person can start a run.

📖 What You’ll Learn

  • Where AI agent costs come from
  • How to set per-run limits
  • How to add a daily spending cap
  • How to rate limit each user

💡 Where LLM Cost Control Begins

Explain Like I’m 5: Pocket Money

Imagine: With a set amount for the week, you can spend freely until it runs out. Then you stop.

What it really means: A budget is a limit that stops the spending.

In testing: The run ends safely when it reaches its token limit.

Cost driverControl
Too many calls in a loopA call limit per run
Very long inputsA token limit, and shorter evidence
Too many runsA rate limit for each user
Many users on one dayA daily spending cap
An expensive model for easy workUsing a cheaper model where it is good enough

🔍 Count What Was Used

Explain Like I’m 5: Counting Your Steps

Imagine: A step counter does not guess, and it shows exactly how far you walked.

What it really means: Use the token counts that the provider returns.

In testing: The adapter adds each call’s real usage to a meter.

Create src/llm/usage.ts.

export class UsageMeter {
  calls = 0;
  inputTokens = 0;
  outputTokens = 0;

  add(inputTokens: number, outputTokens: number): void {
    this.inputTokens += inputTokens;
    this.outputTokens += outputTokens;
  }

  get totalTokens(): number {
    return this.inputTokens + this.outputTokens;
  }

  reset(): void {
    this.calls = 0;
    this.inputTokens = 0;
    this.outputTokens = 0;
  }
}

// Safe as one shared object because the worker handles one run at a time.
export const usageMeter = new UsageMeter();

Now update src/llm/provider-adapter.ts. Add this import.

import { usageMeter } from './usage.js';

Then add this line straight after the response arrives, before the stop_reason check.

usageMeter.add(response.usage.input_tokens, response.usage.output_tokens);

Other providers report usage in their own way, so adapt that line to yours.


🧩 Enforce the Budget

Create src/llm/guarded-llm.ts. It wraps any model client and adds limits and logging, so your agents do not change.

import type * as z from 'zod';
import { log } from '../observability/logger.js';
import type { Metrics } from '../observability/metrics.js';
import type { LlmClient, LlmPrompt } from './llm-client.js';
import type { UsageMeter } from './usage.js';

export class BudgetExceededError extends Error {}

export type BudgetLimits = {
  maxCallsPerRun: number;
  maxTokensPerRun: number;
  maxDailyCostUsd: number;
  inputCostPerMillionUsd: number;
  outputCostPerMillionUsd: number;
};

export interface SpendLedger {
  spentToday(): number;
  add(usd: number): void;
}

export class GuardedLlmClient implements LlmClient {
  readonly providerName: string;

  constructor(
    private readonly inner: LlmClient,
    private readonly meter: UsageMeter,
    private readonly limits: BudgetLimits,
    private readonly ledger: SpendLedger,
    private readonly metrics: Metrics,
    private readonly runId: string,
  ) {
    this.providerName = inner.providerName;
  }

  private checkBudget(): void {
    if (this.meter.calls >= this.limits.maxCallsPerRun) {
      throw new BudgetExceededError('Reached the model call limit for this run');
    }
    if (this.meter.totalTokens >= this.limits.maxTokensPerRun) {
      throw new BudgetExceededError('Reached the token limit for this run');
    }
    if (this.ledger.spentToday() >= this.limits.maxDailyCostUsd) {
      throw new BudgetExceededError('Reached the daily model spending limit');
    }
  }

  async generateStructuredOutput<T>(prompt: LlmPrompt, schema: z.ZodType<T>): Promise<T> {
    this.checkBudget();
    this.meter.calls += 1;

    const started = Date.now();
    const inputBefore = this.meter.inputTokens;
    const outputBefore = this.meter.outputTokens;
    let outcome = 'ok';
    try {
      return await this.inner.generateStructuredOutput(prompt, schema);
    } catch (error) {
      outcome = 'error';
      throw error;
    } finally {
      const inputTokens = this.meter.inputTokens - inputBefore;
      const outputTokens = this.meter.outputTokens - outputBefore;
      const costUsd =
        (inputTokens * this.limits.inputCostPerMillionUsd +
          outputTokens * this.limits.outputCostPerMillionUsd) /
        1_000_000;
      this.ledger.add(costUsd);
      this.metrics.increment('qa_agent_llm_calls_total', { outcome });
      this.metrics.increment('qa_agent_llm_tokens_total', {}, inputTokens + outputTokens);
      // Prompt text is deliberately left out, because it can contain application data.
      log('info', 'llm.call', {
        runId: this.runId,
        provider: this.providerName,
        prompt: prompt.name,
        promptVersion: prompt.version,
        durationMs: Date.now() - started,
        inputTokens,
        outputTokens,
        costUsd,
        outcome,
      });
    }
  }
}

When a limit is reached, the error stops the workflow, the AI agent orchestrator records the reason and a report is still produced.

Set the prices from your provider’s current price list in your configuration. They change, so do not hard-code them.

A cap protects you, but check your provider too

Set a spending limit in your model provider’s account as well. If your code has a bug, that second limit still applies.


🛠️ Add Rate Limits and the Daily Ledger

Explain Like I’m 5: One Turn Each

Imagine: On the slide, everyone gets a turn before anyone goes again.

What it really means: A rate limit stops one person using up everything.

In testing: Each person can start ten runs an hour.

Create src/service/limits.ts.

import type { SpendLedger } from '../llm/guarded-llm.js';
import type { ServiceStore } from './store.js';

export class FixedWindowLimiter {
  private readonly windows = new Map<string, { startedAt: number; count: number }>();

  constructor(
    private readonly limit: number,
    private readonly windowMs: number,
    private readonly now: () => number = Date.now,
  ) {}

  allow(key: string): boolean {
    const time = this.now();
    const current = this.windows.get(key);
    if (!current || time - current.startedAt >= this.windowMs) {
      this.windows.set(key, { startedAt: time, count: 1 });
      return true;
    }
    if (current.count >= this.limit) {
      return false;
    }
    current.count += 1;
    return true;
  }
}

export class StoredSpendLedger implements SpendLedger {
  constructor(private readonly store: ServiceStore) {}

  private key(): string {
    return `spend:${new Date().toISOString().slice(0, 10)}`;
  }

  spentToday(): number {
    return Number(this.store.getSetting(this.key()) ?? 0);
  }

  add(usd: number): void {
    this.store.setSetting(this.key(), String(this.spentToday() + usd));
  }
}

Create tests/unit/budget.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import * as z from 'zod';
import { BudgetExceededError, GuardedLlmClient, type SpendLedger } from '../../src/llm/guarded-llm.js';
import type { LlmClient } from '../../src/llm/llm-client.js';
import { UsageMeter } from '../../src/llm/usage.js';
import { Metrics } from '../../src/observability/metrics.js';
import { FixedWindowLimiter } from '../../src/service/limits.js';

const prompt = { name: 'p', version: '1', task: 't', untrustedData: {} };
const limits = {
  maxCallsPerRun: 2,
  maxTokensPerRun: 1000,
  maxDailyCostUsd: 1,
  inputCostPerMillionUsd: 1000,
  outputCostPerMillionUsd: 2000,
};

function setup(spent = 0) {
  const meter = new UsageMeter();
  let total = spent;
  const ledger: SpendLedger = { spentToday: () => total, add: (usd) => (total += usd) };
  const inner: LlmClient = {
    providerName: 'fake',
    generateStructuredOutput: async <T>(_p: unknown, schema: z.ZodType<T>) => {
      meter.add(100, 50);
      return schema.parse('ok');
    },
  };
  const client = new GuardedLlmClient(inner, meter, limits, ledger, new Metrics(), 'run-1');
  return { client, spent: () => total };
}

test('stops at the call limit', async () => {
  const { client } = setup();
  await client.generateStructuredOutput(prompt, z.string());
  await client.generateStructuredOutput(prompt, z.string());
  await assert.rejects(client.generateStructuredOutput(prompt, z.string()), BudgetExceededError);
});

test('records the cost of each call', async () => {
  const { client, spent } = setup();
  await client.generateStructuredOutput(prompt, z.string());
  assert.equal(spent(), 0.2);
});

test('stops at the daily spending limit', async () => {
  const { client } = setup(1);
  await assert.rejects(client.generateStructuredOutput(prompt, z.string()), /daily/);
});

test('rate limits each user separately, then resets', () => {
  let now = 0;
  const limiter = new FixedWindowLimiter(1, 1000, () => now);
  assert.equal(limiter.allow('asha'), true);
  assert.equal(limiter.allow('asha'), false);
  assert.equal(limiter.allow('ben'), true);
  now = 1000;
  assert.equal(limiter.allow('asha'), true);
});

This rate limiter lives in memory, so it resets on a restart. That is fine for one server. With several servers, keep the counters in a shared store.


📌 Spend Less

TechniqueEffect
Send only the evidence that is neededFewer input tokens
Use a cheaper model for simple stepsLower cost per call
Use your provider’s prompt cachingLower cost for repeated context
Avoid repeating unchanged workFewer calls
Stop as soon as a stage failsNo wasted later stages

Always check quality using your evaluation pack after any change, because a cheaper setup that fails more often may cost more overall.


🌍 Real-Life Example: The Weekend Bill

An agent retries a bad response in a loop. Nobody is watching over the weekend, and the bill arrives on Monday. A call limit per run would have stopped it after a few requests. A daily cap would have been the second line of defence.


🧪 Try It Yourself

  1. Create the files, and run the unit tests.
  2. Set MAX_LLM_CALLS_PER_RUN=1, and run the mock workflow.
  3. Read the stop reason in the report.
  4. Set your provider’s real prices in the configuration.
  5. Calculate the cost of one run.

📋 Ready-to-Copy Prompt

You are an engineer reducing the cost of an LLM-based QA agent.

Per-run data:
[paste calls, tokens and cost by prompt with no sensitive content]

Find the biggest cost drivers. Suggest changes in order of savings, and
say how each one could reduce quality.

Do not invent prices. Tell me to confirm current pricing with my provider.

⚠️ Common Mistakes

  • Relying on people to watch the bill
  • Having no limit on a single run
  • Hard-coding prices that later change
  • Treating a budget stop as a crash
  • Cutting cost without checking quality

🎯 Key Takeaways

  • Limits in code stop mistakes early.
  • Use real token counts from the provider.
  • Combine per-run, per-user and daily limits.
  • Re-evaluate quality after any cost change.

🎤 Interview Questions

1. How do you control LLM costs in an agent?

Model answer: I enforce limits on calls and tokens per run, rate limit users and apply a daily cap. I log usage and cost for each prompt.

2. Why not rely on the provider’s spending limit alone?

Model answer: It is a useful backstop, but it protects the whole account. Per-run limits stop one bad run quickly and explain why.

3. What should happen when a budget is reached?

Model answer: The workflow should stop safely, record the reason and produce a report. It should not fail silently or continue.

4. How would you reduce token usage?

Model answer: I would send only relevant evidence, use cheaper models where evaluation supports it, use caching and avoid repeated work.

5. Why might a cheaper model cost more?

Model answer: If it fails more often, retries and human review can outweigh the savings. I compare the total cost per accepted result.


📝 Assignment

Report the cost for each prompt across five runs. Find the most expensive one, reduce its input and compare quality and cost before and after.


❓ FAQ

What is a sensible daily limit?

Measure a typical run, multiply by your expected number of runs and add a margin.

Does the mock use real tokens?

No. It reports zero, so the token and cost limits need a real provider or a fake client, as in the unit tests.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Production Readiness: Rollout and Incident Runbook.

AI Agent Observability: Logs, LLM Traces and Alerts

When a service fails at 3 a.m., you cannot attach a debugger to yesterday. AI agent observability means recording enough to understand what happened, while keeping sensitive data out. In this lesson you will add structured logs and metrics, then decide which problems deserve an alert.

📖 What You’ll Learn

  • The difference between logs, metrics and traces
  • How to write useful structured logs
  • What to record about each model call
  • Which alerts are worth having

💡 The Three Parts of AI Agent Observability

Explain Like I’m 5: A Doctor’s Tools

Imagine: A doctor reads your notes, checks your temperature and follows how your symptoms developed.

What it really means: Logs, metrics and traces each show something different.

In testing: Logs give detail, metrics show trends and traces follow one run.

SignalAnswersExample
LogsWhat exactly happened?A model call failed for a run
MetricsHow is it trending?Failed runs per hour
TracesWhere did the time go?Which stage of a run was slow

🔍 Write Structured Logs

Explain Like I’m 5: Filling in a Form

Imagine: A form is easy to search. A long story about the same event is not.

What it really means: Structured logs use named fields, so that they can be searched.

In testing: Find every event for one run ID in seconds.

Create src/observability/logger.ts.

import { redact } from '../core/guardrails.js';

export function log(
  level: 'info' | 'warn' | 'error',
  event: string,
  fields: Record<string, unknown> = {},
): void {
  const line = JSON.stringify({ at: new Date().toISOString(), level, event, ...fields });
  console.log(redact(line));
}

Every line is JSON, and every line passes through the redact guardrail. Always include a runId when one exists, so that you can follow a single run from start to finish.


🧩 Record Metrics

Create src/observability/metrics.ts.

export class Metrics {
  private readonly counters = new Map<string, number>();

  increment(name: string, labels: Record<string, string> = {}, by = 1): void {
    const text = Object.entries(labels)
      .map(([key, value]) => `${key}="${value}"`)
      .join(',');
    const series = text ? `${name}{${text}}` : name;
    this.counters.set(series, (this.counters.get(series) ?? 0) + by);
  }

  render(): string {
    return [...this.counters].map(([series, value]) => `${series} ${value}`).join('\n') + '\n';
  }
}

The output uses a simple text format that monitoring tools such as Prometheus can read. In our test run, /metrics returned this.

qa_agent_llm_calls_total{outcome="ok"} 3
qa_agent_llm_tokens_total 0
qa_agent_runs_total{status="completed"} 1

The token count is zero because the mock model uses no tokens.


🛠️ Record Every Model Call

Explain Like I’m 5: A Taxi Receipt

Imagine: The receipt shows the time, distance and price. It does not record what you talked about.

What it really means: Log facts about each model call and leave out the content.

In testing: Record the model, prompt version, tokens and duration.

The wrapper you will build in the next lesson writes one log line for each call.

{
  "level": "info",
  "event": "llm.call",
  "runId": "run-ccd886e6",
  "provider": "mock",
  "prompt": "parse-requirement",
  "promptVersion": "1",
  "durationMs": 2,
  "inputTokens": 0,
  "outputTokens": 0,
  "costUsd": 0,
  "outcome": "ok"
}

Be careful with prompts and responses

They can contain page text, requirements or secrets. Our logs leave them out. If you need them for debugging, turn this on deliberately, restrict access and set a short retention period.

As you grow, consider OpenTelemetry, which is a widely used standard for traces and metrics. It has conventions for generative AI calls, so check the current status of those conventions before depending on them. Your own agent’s run state already acts as a simple trace.


📌 Choose Your Alerts

An alert should mean that a person needs to act. Too many alerts, and people stop paying attention.

AlertWhy it matters
Service health check failsNobody can use it
No successful run in 24 hoursIt may have silently stopped
Internal failure rate is highSomething is broken
Daily spending is near its limitCost is out of line
A worker is stuck on one jobThe queue has stopped moving
Approval waiting too longA person needs a reminder

🔑 Build a Simple Dashboard

PanelPurpose
Runs by statusOverall health
Model calls and errorsProvider problems
Tokens and cost per daySpending
Time spent waiting for approvalTeam bottlenecks
Evaluation pass rate over timeQuality drift

The AI Test Report explains one run. A dashboard shows the pattern across many.


🌍 Real-Life Example: The Quiet Failure

A service reports no errors for a week, and everyone is happy. In fact, a configuration change stopped all runs from starting. There were no errors because nothing happened. An alert for “no successful run in 24 hours” would have found it on day one.


🧪 Try It Yourself

  1. Create both files.
  2. Use log in a short script, with a run ID.
  3. Add a password to a log field, and confirm that it is removed.
  4. Increment a counter and print render().
  5. Write down your three most important alerts.

📋 Ready-to-Copy Prompt

You are a site reliability engineer designing monitoring for a QA agent
service.

Available signals:
[list your logs and metrics with no secrets]

Suggest a small set of alerts. For each one, give the condition, why a
person must act and the first thing that person should check.

Do not suggest any alert that nobody would act on.

⚠️ Common Mistakes

  • Logging free text that cannot be searched
  • Writing full prompts and page content to logs
  • Alerting on everything
  • Only monitoring errors, and missing silence
  • Leaving out the run ID

🎯 Key Takeaways

  • Logs, metrics and traces answer different questions.
  • Use structured logs with a run ID.
  • Record facts about model calls, not their content.
  • Alert only when a person needs to act.

🎤 Interview Questions

1. What is the difference between logs, metrics and traces?

Model answer: Logs record individual events, metrics show numbers over time and traces follow one request through its steps.

2. What would you log for each LLM call?

Model answer: I would log the run ID, provider, prompt version, duration, token counts, cost and outcome. I would leave out sensitive content by default.

3. Why is logging prompts risky?

Model answer: Prompts can contain sensitive application data or credentials. Logs are often widely accessible and kept for a long time.

4. How would you detect a service that has silently stopped?

Model answer: I would alert on the absence of successful runs, because a stopped system produces no errors.

5. What makes an alert good?

Model answer: It is actionable, it is rare enough to be taken seriously and it tells the responder where to start.


📝 Assignment

Add a metric for approval waiting time. Then design a one-page dashboard and write the first three steps a person should take for each alert.


❓ FAQ

Do I need an observability platform?

Not at first. Structured logs and a few metrics go a long way.

Should the metrics endpoint be public?

No. In this project, only an admin token can read it.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Cost Control: Token Budgets and Rate Limits for AI Agents.

Prompt Versioning and Evaluation Gates: CI/CD for AI Agents

A single changed sentence in a prompt can make your agent worse, and no unit test will notice. Prompt versioning treats prompts as code, so each change is reviewed and traceable. In this lesson you will add an evaluation gate that blocks bad releases, and a release pipeline that a person must approve.

📖 What You’ll Learn

  • Why prompts need versions
  • What makes up an agent release
  • How to block a release automatically
  • How to roll back a bad change

💡 Why Prompt Versioning Matters

Explain Like I’m 5: Changing a Recipe

Imagine: Add a little more salt and the cake tastes wrong. If you did not write down the change, you cannot go back.

What it really means: Each prompt change must be recorded.

In testing: Every model call logs which prompt version it used.

Your prompts already have a version field, and they live in Git. Follow two simple rules:

  1. Change the version number whenever you change a prompt’s meaning.
  2. Change prompts through pull requests, just like code.

🔍 What Is an Agent Release?

Explain Like I’m 5: A Band’s Line-Up

Imagine: If a new drummer joins, the band sounds different, even with the same songs.

What it really means: A release includes more than the code.

In testing: A new model can change results without any code change.

PartTracked by
Application codeGit commit and image tag
PromptsVersion number and Git history
ModelLLM_MODEL setting
DependenciesLockfile
Tool definitionsPinned package version

Run the evaluation pack whenever any of these change.


🧩 Build the Evaluation Gate

Explain Like I’m 5: The Height Sign at a Ride

Imagine: “You must be this tall.” The sign does not care how excited you are.

What it really means: A gate applies fixed rules to every release.

In testing: One safety failure blocks a release, whatever else improved.

Create src/evaluation/gate.ts.

export type EvalSummary = {
  release: string;
  passRate: number;
  safetyFailures: number;
  averageCostUsd: number;
};

export type GateLimits = { maxPassRateDrop: number; maxCostIncreasePercent: number };

export function evaluateGate(
  baseline: EvalSummary,
  candidate: EvalSummary,
  limits: GateLimits = { maxPassRateDrop: 5, maxCostIncreasePercent: 25 },
): { pass: boolean; reasons: string[] } {
  const reasons: string[] = [];

  if (candidate.safetyFailures > 0) {
    reasons.push(`${candidate.safetyFailures} safety failure(s)`);
  }
  const drop = baseline.passRate - candidate.passRate;
  if (drop > limits.maxPassRateDrop) {
    reasons.push(`Pass rate dropped by ${drop} points`);
  }
  if (baseline.averageCostUsd > 0) {
    const increase = (candidate.averageCostUsd / baseline.averageCostUsd - 1) * 100;
    if (increase > limits.maxCostIncreasePercent) {
      reasons.push(`Cost rose by ${Math.round(increase)}%`);
    }
  }
  return { pass: reasons.length === 0, reasons };
}

Create tests/unit/gate.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { evaluateGate } from '../../src/evaluation/gate.js';

const baseline = { release: 'r1', passRate: 90, safetyFailures: 0, averageCostUsd: 0.2 };

test('an equal or better release passes', () => {
  assert.equal(evaluateGate(baseline, { ...baseline, release: 'r2', passRate: 92 }).pass, true);
});

test('any safety failure blocks the release', () => {
  const result = evaluateGate(baseline, { ...baseline, passRate: 100, safetyFailures: 1 });
  assert.deepEqual(result, { pass: false, reasons: ['1 safety failure(s)'] });
});

test('a large drop in quality or rise in cost blocks the release', () => {
  assert.equal(evaluateGate(baseline, { ...baseline, passRate: 80 }).pass, false);
  assert.equal(evaluateGate(baseline, { ...baseline, averageCostUsd: 0.3 }).pass, false);
});

Create scripts/eval-gate.ts, so that a pipeline can run the check.

import { readFile } from 'node:fs/promises';
import { evaluateGate, type EvalSummary } from '../src/evaluation/gate.js';

const read = async (path: string) => JSON.parse(await readFile(path, 'utf8')) as EvalSummary;
const [baselinePath, candidatePath] = process.argv.slice(2);

const result = evaluateGate(await read(baselinePath), await read(candidatePath));
console.log(result.pass ? 'Evaluation gate passed' : `Evaluation gate failed: ${result.reasons.join('; ')}`);
process.exit(result.pass ? 0 : 1);

Each file contains a summary of scored results, like this one.

{
  "release": "2026-11-10-prompt-v2",
  "passRate": 90,
  "safetyFailures": 0,
  "averageCostUsd": 0.2
}

Store evaluation/baseline.json for your current release. Because people score your evaluation pack, produce evaluation/candidate.json from that reviewed work, and do not generate it with a model.

Numbers can hide a bad release

Averages can look fine while one important case gets worse. Read the individual failures before you approve, even when the gate passes.


🛠️ Create the Release Pipeline

Create .github/workflows/release.yml.

name: Release

on:
  workflow_dispatch:

permissions:
  contents: read
  packages: write

jobs:
  evaluate:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test:unit
      # evaluation/candidate.json comes from your reviewed evaluation run.
      - run: npx tsx scripts/eval-gate.ts evaluation/baseline.json evaluation/candidate.json

  publish:
    needs: evaluate
    runs-on: ubuntu-latest
    # Set required reviewers on this environment in your repository settings.
    environment: production
    steps:
      - uses: actions/checkout@v4
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
StepPurpose
workflow_dispatchA person starts each release
evaluate jobCode checks and the evaluation gate
needs: evaluateNo image is built if the gate fails
environment: productionRequires your named reviewers
Tag with github.shaEvery image traces back to a commit

In your repository settings, add required reviewers to the production environment. Your team’s policy may also require pinning third-party actions to a full commit hash.


📌 Roll Back Safely

What changedHow to roll back
Code or promptsRedeploy the previous image
ModelRestore the previous LLM_MODEL
DatabaseKeep changes backward compatible

Make database changes in two steps: add new columns first, and remove old ones in a later release. An older image can then still run. You learned to compare results over time in LLM Regression Testing in CI.


🌍 Real-Life Example: The Friendly Rewrite

Someone rewrites a planner prompt to be “friendlier”. The pass rate stays the same, but a safety case now fails, because the planner no longer refuses invented requirements. The gate blocks the release, and the team finds the sentence that caused it.


🧪 Try It Yourself

  1. Create the files and run the unit tests.
  2. Create two summary files, and run the gate.
  3. Add a safety failure, and confirm that the script exits with 1.
  4. Change a prompt, and increase its version.
  5. Write down what a rollback would look like.

📋 Ready-to-Copy Prompt

You are a release manager reviewing a change to an AI agent.

What changed: [prompt, model, code or tools]
Baseline results: [paste summary]
Candidate results: [paste summary]
Individual failures: [paste with no secrets]

Tell me whether you would release it, what looks risky even if the numbers
pass and what I should check by hand. Do not change any numbers.

⚠️ Common Mistakes

  • Editing prompts directly on a server
  • Forgetting that a model change is a release
  • Releasing when an evaluation has not run
  • Using an average to hide a safety failure
  • Making database changes that cannot be rolled back

🎯 Key Takeaways

  • Prompts are code, so version and review them.
  • A release includes code, prompts, model and tools.
  • Automatic gates block known regressions.
  • Practise rolling back before you need to.

🎤 Interview Questions

1. How do you version prompts?

Model answer: I store them in Git with a version number and change them through pull requests. Each model call logs the version, so results stay traceable.

2. What is an evaluation gate?

Model answer: It is an automatic check that compares a candidate release with a baseline. It blocks the release if quality falls, cost rises too far or there is a safety failure.

3. Why treat a model change as a release?

Model answer: A different model can behave differently with the same prompt. It needs the same evaluation as a code change.

4. How do you roll back an AI agent?

Model answer: I redeploy the previous image and restore its model setting. Backward-compatible database changes make that safe.

5. Should a gate replace human review?

Model answer: No. It catches known regressions, while people look at individual failures and at things the metrics do not capture.


📝 Assignment

Create baseline and candidate summaries from your evaluation pack. Add a rule that blocks a release if average review time rises by more than 20%, and add a unit test.


❓ FAQ

Should prompts live in a separate tool?

Git is a good start. A prompt management tool can help when non-engineers need to edit prompts.

How often should I run the evaluation?

Run it for each release candidate, and on a schedule, because a provider can change a model’s behaviour.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Observability: Logs, LLM Traces and Alerts.

Human Approval Workflow for AI Agents: Slack, Jira and Pull Requests

Approvals work only if people see them and can act on them easily. A human approval workflow brings the decision to where your team already works, without weakening the rules. In this lesson you will add Slack buttons with verified signatures, and learn when a pull request is the better place to approve.

📖 What You’ll Learn

  • Which approval channel suits each decision
  • How to verify that a request really came from Slack
  • How to prevent replayed requests
  • Why generated code should be reviewed in a pull request

💡 Choose the Right Human Approval Workflow

Explain Like I’m 5: Different Signatures

Imagine: A parent can sign a school trip form quickly. Buying a house needs a lawyer to read every page.

What it really means: Match the depth of review to the risk.

In testing: A quick button may suit a test run, but code needs a proper review.

ChannelBest forLimit
API or web pageAny decision, with full detailPeople must remember to look
SlackQuick, low-risk approvalsLittle room to review
JiraDecisions that need a ticket trailSlower
Pull requestReviewing generated codeNeeds a code reviewer

A pull request gives you a diff, comments and branch protection, so it suits generated tests best. You will open one in Lesson 89.


🔍 Trust Only Signed Requests

Explain Like I’m 5: A Wax Seal

Imagine: A king’s letter carries a wax seal. If the seal is broken or wrong, nobody trusts the letter.

What it really means: A signature proves that the request came from Slack.

In testing: Anyone can send a request, but only Slack can sign it correctly.

Your webhook must be reachable by Slack, so anyone on the internet could also try to call it. Without verification, a stranger could send a fake “approve” click.

Slack signs each request using your signing secret. You rebuild the signature from the raw body and compare the two. Create src/integrations/slack.ts.

import { createHmac, timingSafeEqual } from 'node:crypto';
import express, { type Express } from 'express';
import { recordDecision, type ApprovalNotifier } from '../service/stored-approver.js';
import type { ServiceStore } from '../service/store.js';

export function verifySlackSignature(input: {
  secret: string;
  timestamp: string;
  signature: string;
  rawBody: string;
  nowSeconds: number;
}): boolean {
  const age = Math.abs(input.nowSeconds - Number(input.timestamp));
  if (!Number.isFinite(age) || age > 300) {
    return false;
  }
  const expected = `v0=${createHmac('sha256', input.secret)
    .update(`v0:${input.timestamp}:${input.rawBody}`)
    .digest('hex')}`;
  const a = Buffer.from(expected);
  const b = Buffer.from(input.signature);
  return a.length === b.length && timingSafeEqual(a, b);
}

export type SlackAction = { slackUser: string; approvalId: string; approved: boolean };

export function parseSlackAction(rawBody: string): SlackAction | null {
  try {
    const payload = JSON.parse(new URLSearchParams(rawBody).get('payload') ?? '');
    const [decision, approvalId] = String(payload.actions[0].value).split(':');
    if (!approvalId || !['approve', 'reject'].includes(decision)) {
      return null;
    }
    return { slackUser: String(payload.user.id), approvalId, approved: decision === 'approve' };
  } catch {
    return null;
  }
}

export function mountSlackRoutes(
  app: Express,
  store: ServiceStore,
  secret: string,
  approvers: Record<string, string>,
): void {
  app.post(
    '/webhooks/slack',
    express.raw({ type: 'application/x-www-form-urlencoded', limit: '64kb' }),
    (request, response) => {
      const rawBody = Buffer.isBuffer(request.body) ? request.body.toString('utf8') : '';
      const trusted = verifySlackSignature({
        secret,
        rawBody,
        timestamp: request.header('x-slack-request-timestamp') ?? '',
        signature: request.header('x-slack-signature') ?? '',
        nowSeconds: Math.floor(Date.now() / 1000),
      });
      if (!trusted) {
        response.status(401).send('Invalid signature');
        return;
      }
      const action = parseSlackAction(rawBody);
      const actorId = action ? approvers[action.slackUser] : undefined;
      if (!action || !actorId) {
        response.json({ text: 'You are not registered as an approver.' });
        return;
      }
      const result = recordDecision(store, action.approvalId, actorId, action.approved);
      response.json({ text: result === 'ok' ? 'Decision recorded.' : `Not recorded: ${result}` });
    },
  );
}

export function createSlackNotifier(webhookUrl: string): ApprovalNotifier {
  return async (approval) => {
    const button = (text: string, decision: string, style: string) => ({
      type: 'button',
      text: { type: 'plain_text', text },
      style,
      value: `${decision}:${approval.id}`,
    });
    const response = await fetch(webhookUrl, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        text: `Approval needed: ${approval.action}`,
        blocks: [
          {
            type: 'section',
            text: {
              type: 'mrkdwn',
              text: `*${approval.action}* for run \`${approval.run_id}\`. Review the details in the service first.`,
            },
          },
          {
            type: 'actions',
            elements: [
              button('Approve', 'approve', 'primary'),
              button('Reject', 'reject', 'danger'),
            ],
          },
        ],
      }),
    });
    if (!response.ok) {
      throw new Error(`Slack returned ${response.status}`);
    }
  };
}
CheckWhat it stops
HMAC signatureForged requests
Five-minute windowOld requests being replayed
timingSafeEqualGuessing a signature by timing
Raw body, before parsingSignature mismatches
Slack user mappingUnknown people approving
recordDecisionSelf-approval and double decisions

Keep the message short and safe

The Slack message links to the run and does not include the generated code or any page text. Application content could contain secrets or hostile links.


🧩 Test the Signature Check

Explain Like I’m 5: The Fake Seal

Imagine: You try sealing a letter with a potato stamp, just to check that the guards notice.

What it really means: Test with changed and expired requests.

In testing: A changed body and an old timestamp must both fail.

Create tests/unit/slack.test.ts.

import assert from 'node:assert/strict';
import { createHmac } from 'node:crypto';
import { test } from 'node:test';
import { parseSlackAction, verifySlackSignature } from '../../src/integrations/slack.js';

const secret = 'test-signing-secret';
const rawBody = `payload=${encodeURIComponent(
  JSON.stringify({ user: { id: 'U1' }, actions: [{ value: 'approve:apr-1' }] }),
)}`;
const nowSeconds = 1_800_000_000;
const sign = (timestamp: number, body: string) =>
  `v0=${createHmac('sha256', secret).update(`v0:${timestamp}:${body}`).digest('hex')}`;
const check = (timestamp: number, signature: string) =>
  verifySlackSignature({ secret, timestamp: String(timestamp), signature, rawBody, nowSeconds });

test('accepts a correctly signed request', () => {
  assert.equal(check(nowSeconds, sign(nowSeconds, rawBody)), true);
});

test('rejects a changed body', () => {
  assert.equal(check(nowSeconds, sign(nowSeconds, `${rawBody}x`)), false);
});

test('rejects an old request, even with a valid signature', () => {
  const old = nowSeconds - 301;
  assert.equal(check(old, sign(old, rawBody)), false);
});

test('reads the decision from the button', () => {
  assert.deepEqual(parseSlackAction(rawBody), { slackUser: 'U1', approvalId: 'apr-1', approved: true });
  assert.equal(parseSlackAction('payload=not-json'), null);
});

🛠️ Connect It

In your Slack app settings, turn on interactivity and set the request URL to your service’s /webhooks/slack address. Then add these settings, using your own values.

SLACK_SIGNING_SECRET=your-signing-secret
SLACK_WEBHOOK_URL=your-incoming-webhook-url
SLACK_APPROVERS={"U012ABC":"ben"}

SLACK_APPROVERS maps a Slack user to a person in your service. Anyone who is not listed is refused, even if they can see the channel. Expose only this single path through your proxy. In src/service/main.ts, pass the notifier and mount the route, as shown in the final file in Lesson 88.


📌 Handling Slow Decisions

Our approval waits up to 60 minutes and then counts as a no. Choose a time that suits your team. A rejected or expired run stops safely, and it can be requested again.

The worker waits while an approval is pending, which pauses the queue. That is acceptable for one team. If it becomes a bottleneck, save the run’s state, free the worker and resume after the decision.


🌍 Real-Life Example: The Rubber Stamp

A team gets dozens of approval messages a day, and people start clicking without reading. A generated test that does nothing gets approved. Moving code review to pull requests, and keeping Slack for small decisions, brings real attention back. You saw this risk in MCP Security for Testers.


🧪 Try It Yourself

  1. Create the files and run the unit tests.
  2. Change one character of a signature in a test.
  3. Add a test for a missing timestamp.
  4. Map a test user in SLACK_APPROVERS.
  5. Decide which channel suits each of your approvals.

📋 Ready-to-Copy Prompt

You are a security engineer reviewing a webhook that approves AI agent
actions.

Code:
[paste the webhook with no secrets]

Check signature verification, replay protection, raw body handling, user
mapping and what happens with malformed input.

Confirm the current signing rules against the vendor's official docs, and
write a negative test for each problem that you find.

⚠️ Common Mistakes

  • Trusting a webhook without checking its signature
  • Parsing the body before verifying it
  • Letting anyone in a channel approve
  • Sending code or page text to a chat channel
  • Asking for so many approvals that nobody reads them

🎯 Key Takeaways

  • Match the approval channel to the risk.
  • Verify every webhook using its signature.
  • Only mapped people can approve.
  • Generated code deserves a real review.

🎤 Interview Questions

1. How do you verify a Slack webhook?

Model answer: I recompute an HMAC of the version, timestamp and raw body using the signing secret. I compare it in constant time, and reject old timestamps.

2. What is a replay attack?

Model answer: An attacker resends a valid captured request. A timestamp window limits how long a captured request can work.

3. Why is the raw body needed?

Model answer: The signature covers the exact bytes that were sent. Parsing and re-serialising can change them, so the signature would not match.

4. Where should generated code be approved?

Model answer: In a pull request, because it provides a diff, comments and branch protection. A chat button does not support a proper code review.

5. How do you prevent approval fatigue?

Model answer: I ask only for decisions that matter, make each request specific and move detailed reviews to a suitable tool.


📝 Assignment

Write a table of every action your agent can take, its risk and the approval channel it needs. Then add a test which proves that an unmapped Slack user cannot approve anything.


❓ FAQ

Can I use Microsoft Teams?

Yes. The idea is the same, but its signing and message formats differ, so follow its documentation.

Does Slack need to reach my private network?

Yes, for buttons to work. Expose only the webhook path through your proxy, or use approvals through the API instead.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Prompt Versioning and Evaluation Gates: CI/CD for AI Agents.

AI Agent Access Control: Secrets, Sign-In, Roles and Network Limits

A service that anyone can call is a service that anyone can misuse. AI agent access control decides who can do what, and it must be enforced by your code. In this lesson you will replace the temporary local security with real tokens and roles, then test that the limits actually hold.

📖 What You’ll Learn

  • How to store API tokens safely
  • How roles limit what people can do
  • How to manage secrets in production
  • What network limits to apply

💡 Three Layers of AI Agent Access Control

Explain Like I’m 5: A Building with Security

Imagine: A guard checks your badge at the door. Your badge opens only certain rooms. The building also has a fence around it.

What it really means: Sign-in, roles and network limits work together.

In testing: A valid token still cannot reach a role it does not have.

LayerQuestionOur answer
AuthenticationWho are you?A bearer token
AuthorisationWhat may you do?Roles and permissions
NetworkCan you reach it at all?VPN and firewall rules

🔍 Roles and Permissions

The roles are in src/service/security.ts, which you created in AI Agent as a Service.

RoleCan read runsCan start runsCan approveCan operate
ViewerYesNoNoNo
RequesterYesYesNoNo
ApproverYesNoYesNo
AdminYesYesYesYes

Give people the lowest role that lets them do their job. Keep admin rare.


🧩 Store Tokens Safely

Explain Like I’m 5: The Coat Check

Imagine: You get a numbered ticket for your coat. The ticket alone proves that the coat is yours, so the cloakroom only needs a matching number.

What it really means: The service stores a fingerprint and never the token itself.

In testing: A stolen database does not reveal usable tokens.

Create src/service/token-security.ts.

import { createHash, randomBytes } from 'node:crypto';
import type { Request } from 'express';
import type { Actor, Role, Security } from './security.js';
import type { ServiceStore } from './store.js';

export const hashToken = (token: string) => createHash('sha256').update(token).digest('hex');

export function issueToken(store: ServiceStore, name: string, role: Role): string {
  const token = `qat_${randomBytes(32).toString('base64url')}`;
  store.addToken(name, role, hashToken(token));
  return token;
}

export class TokenSecurity implements Security {
  constructor(private readonly store: ServiceStore) {}

  authenticate(request: Request): Actor | null {
    const header = request.header('authorization') ?? '';
    const match = header.match(/^Bearer (\S+)$/);
    if (!match) {
      return null;
    }
    const token = this.store.findToken(hashToken(match[1]));
    if (!token || token.revoked_at !== null) {
      return null;
    }
    return { id: token.name, role: token.role as Role };
  }
}

Long random tokens are safe to fingerprint with SHA-256. Passwords are different, and they need a slow password-hashing function.

Create scripts/create-token.ts.

import { loadServiceConfig } from '../src/service/config.js';
import type { Role } from '../src/service/security.js';
import { ServiceStore } from '../src/service/store.js';
import { issueToken } from '../src/service/token-security.js';

const [name, role] = process.argv.slice(2);
if (!name || !['viewer', 'requester', 'approver', 'admin'].includes(role)) {
  console.error('Usage: npm run create-token -- <name> <viewer|requester|approver|admin>');
  process.exit(1);
}

const store = new ServiceStore(loadServiceConfig().DATABASE_PATH);
console.log(issueToken(store, name, role as Role));
console.error('Save this token now. Only its hash is stored, so it cannot be shown again.');
store.close();

Add the script and create one token per person, and not one shared by everyone.

npm pkg set scripts.create-token="tsx scripts/create-token.ts"
npm run create-token -- asha requester
npm run create-token -- ben approver

Now use it. In src/service/main.ts, import TokenSecurity and replace the security option.

security: config.AUTH_MODE === 'token' ? new TokenSecurity(store) : new LocalDevSecurity(),

🛠️ Test the Limits

Explain Like I’m 5: Trying the Locked Door

Imagine: After locking up, you push the door to be sure that it really is locked.

What it really means: Prove that each forbidden action is refused.

In testing: A viewer who tries to start a run must get a 403.

Create tests/unit/security.test.ts. It starts the real API.

import assert from 'node:assert/strict';
import type { Server } from 'node:http';
import type { AddressInfo } from 'node:net';
import { after, before, test } from 'node:test';
import { createApi } from '../../src/service/api.js';
import { ServiceStore } from '../../src/service/store.js';
import { issueToken, TokenSecurity } from '../../src/service/token-security.js';

const store = new ServiceStore(':memory:');
const viewer = issueToken(store, 'vic', 'viewer');
const requester = issueToken(store, 'asha', 'requester');
const approver = issueToken(store, 'ben', 'approver');
let server: Server;
let url: string;

before(async () => {
  const app = createApi({ root: '.', store, security: new TokenSecurity(store), environments: ['demo'] });
  await new Promise<void>((done) => {
    server = app.listen(0, '127.0.0.1', () => done());
  });
  url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
after(() => server.close());

const post = (path: string, token: string | null, body: unknown) =>
  fetch(url + path, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      ...(token ? { Authorization: `Bearer ${token}` } : {}),
    },
    body: JSON.stringify(body),
  });
const run = { requirement: 'Login', environment: 'demo' };

test('requests without a valid token are refused', async () => {
  assert.equal((await post('/v1/runs', null, run)).status, 401);
  assert.equal((await post('/v1/runs', 'qat_wrong', run)).status, 401);
});

test('roles limit what each person can do', async () => {
  assert.equal((await post('/v1/runs', viewer, run)).status, 403);
  assert.equal((await post('/v1/runs', approver, run)).status, 403);
  assert.equal((await post('/v1/runs', requester, run)).status, 202);
});

test('only approved environments are accepted', async () => {
  const response = await post('/v1/runs', requester, { requirement: 'Login', environment: 'production' });
  assert.equal(response.status, 400);
});

test('bad input gets a clean error', async () => {
  const response = await fetch(`${url}/v1/runs`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${requester}` },
    body: '{broken',
  });
  assert.equal(response.status, 400);
});

test('only the hash of a token is stored, and it can be revoked', async () => {
  assert.equal(store.findToken(requester), undefined);
  const temporary = issueToken(store, 'temp', 'requester');
  store.revokeToken('temp');
  assert.equal((await post('/v1/runs', temporary, run)).status, 401);
});

Test what must fail

A security feature is proven by what it refuses. Your access tests should mostly be about the people and actions that must not succeed.


📌 Handle Secrets

SecretWhere it belongs
Model API keySecrets manager or protected server file
Slack signing secretSecrets manager or protected server file
Test account passwordSecrets manager, and synthetic only
API tokensOnly hashes stored, and shown once

Rotate secrets on a schedule, and immediately after any suspected leak. Use a separate key for each environment. The MCP Security for Testers lesson covers the same principles.


🔑 Limit the Network

  • Allow inbound traffic only from your VPN or proxy.
  • Allow outbound traffic only to your test environment and model provider.
  • Block access to cloud metadata addresses, such as 169.254.169.254.
  • Keep your database and artifact storage private.

Your application has its own allowlist too. Requests can name only configured environments, and the explorer checks every host.


💡 When to Add Single Sign-On

Tokens are enough to start. As more people use the service, connect it to your company’s identity provider, using OpenID Connect, so that leavers lose access automatically. Your Security interface means that only one class has to change.


🌍 Real-Life Example: The Shared Token

A team shares one admin token in a chat. Months later, someone approves a risky run, and the audit log says only “admin”. Nobody can tell who it was. One token per person, with the lowest role needed, makes every action traceable.


🧪 Try It Yourself

  1. Create the files, and run the unit tests.
  2. Create a token for each of the four roles.
  3. Try each API route with each token.
  4. Revoke a token, and confirm that it stops working.
  5. Search your logs for any token value.

📋 Ready-to-Copy Prompt

You are an application security engineer reviewing access control for a QA
agent service.

Routes and permissions:
[paste them with no secrets]

Find any route that is missing a check, any role that has too much power
and anything that could let one person both request and approve an action.

Write a negative test case for each finding.

⚠️ Common Mistakes

  • Sharing one token across a team
  • Storing tokens in plain text
  • Giving everyone the admin role
  • Testing only the allowed cases
  • Letting the agent’s server reach anything on the internet

🎯 Key Takeaways

  • Authentication, roles and network limits work together.
  • Store only hashes of long random tokens.
  • Give each person the lowest role they need.
  • Prove your security with negative tests.

🎤 Interview Questions

1. What is the difference between authentication and authorisation?

Model answer: Authentication confirms who someone is. Authorisation decides what they are allowed to do.

2. How would you store API tokens?

Model answer: I would generate long random values, show them once and store only a hash. Tokens should be individual and revocable.

3. How would you test role-based access?

Model answer: I would test every role against every route, focusing on actions that must be refused. I would also test missing, invalid and revoked tokens.

4. Why block cloud metadata addresses?

Model answer: They can expose instance credentials. If an agent or test can reach them, a mistake could leak access to the cloud account.

5. Why does every action need an individual identity?

Model answer: It makes the audit trail meaningful, and it allows one person’s access to be removed without affecting others.


📝 Assignment

Create a matrix of every route against every role, with the expected status code. Automate it as a single data-driven test, and include a revoked token.


❓ FAQ

Can I use this token approach permanently?

For a small internal tool, it can work well. Larger organisations usually prefer single sign-on.

Why SHA-256 for tokens but not for passwords?

Long random tokens cannot be guessed. Passwords can be, so they need a slow function such as Argon2.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Human Approval Workflow for AI Agents: Slack, Jira and Pull Requests.

Deploy an AI Agent to the Cloud: A Safe First Release

There are many ways to deploy an AI agent, and most are more complicated than a first release needs. This lesson uses one small cloud server running one container. That setup suits your SQLite design, keeps costs predictable and is easy to understand. The ideas apply on any cloud provider.

📖 What You’ll Learn

  • Which hosting option suits this agent
  • What persistent storage you need
  • How to keep the service off the public internet
  • How to point it at your own test environment

💡 Choose Where to Deploy an AI Agent

Explain Like I’m 5: Choosing a Home for a Fish

Imagine: A goldfish needs a clean bowl, and it does not need the ocean.

What it really means: Choose the simplest hosting that meets your needs.

In testing: One small server is enough for one team.

OptionGood forWatch out for
One virtual serverThis first releaseYou maintain the server
Managed container serviceLess server workLong jobs and storage limits
KubernetesMany services and teamsMuch more to learn and run

Serverless platforms often limit run time and may lose local files, so they fit this worker poorly. The agent’s runs are long, and they need a browser and a persistent disk.


🔍 Plan the Server

Browsers need plenty of memory. Start with about 2 CPUs and 4 GB of RAM, then adjust after you have measured real runs.

NeedOur answer
Persistent dataNamed Docker volumes
Private accessBind to 127.0.0.1 behind a VPN or proxy
HTTPSA reverse proxy in front
SecretsA protected file outside the image
UpdatesA new image version for each release

🧩 Define the Service

Explain Like I’m 5: A Moving-In List

Imagine: When you move house, you list which boxes go in which room. Nothing gets forgotten.

What it really means: A Compose file lists exactly how the container should run.

In testing: It names the volumes, limits and private port.

Create compose.yaml.

services:
  qa-agent:
    image: ghcr.io/your-org/qa-agent:${RELEASE:?Set RELEASE to an image version}
    restart: unless-stopped
    init: true
    env_file: /etc/qa-agent/agent.env
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - agent-data:/app/data
      - agent-artifacts:/app/artifacts
      - agent-reports:/app/reports
    security_opt:
      - no-new-privileges:true
    mem_limit: 4g
    cpus: 2
    shm_size: 1gb

volumes:
  agent-data:
  agent-artifacts:
  agent-reports:
SettingWhy
RELEASE is requiredYou always state which version runs
127.0.0.1:8080Nothing is exposed directly to the internet
VolumesData survives when the container is replaced
no-new-privilegesThe process cannot gain extra rights
mem_limit and cpusA browser cannot starve the whole server
shm_sizeChromium needs shared memory

🛠️ Store Secrets on the Server

Create /etc/qa-agent/agent.env on the server, and let only root read it.

sudo mkdir -p /etc/qa-agent
sudo touch /etc/qa-agent/agent.env
sudo chmod 600 /etc/qa-agent/agent.env

Add your settings to that file, after reading AI Agent Access Control. Use AUTH_MODE=token, and never local-dev.


📌 Keep It Private

Explain Like I’m 5: A Door with No Street

Imagine: A room with no door to the street can only be reached from inside the house.

What it really means: The service is reachable only through your private network.

In testing: Approvers connect using the company VPN.

Run the service on a private network, or behind a VPN, whenever you can. If it must be reachable from outside, place a reverse proxy in front to provide HTTPS. Only your proxy and your VPN should be able to reach port 8080.

Limit where the agent can go

Configure the server’s firewall so that outbound traffic reaches only your test environment and your model provider. Even if something goes wrong, it then has few places to go.


🔑 Deploy It

export RELEASE=your-image-version
docker compose up -d
docker compose ps
curl http://127.0.0.1:8080/readyz

💡 Point It at Your Test Environment

The image starts the local demo so that you can prove the deployment first. Only use a real test environment once you are authorised. Then make these changes:

  1. Set BASE_URL to your approved test environment.
  2. Remove the webServer block from playwright.config.ts.
  3. Change the Dockerfile’s last line so that it starts only the service.
  4. Replace 'demo' in the environments list with your environment’s name.

🔍 Roll Back

Explain Like I’m 5: Keeping the Old Key

Imagine: You keep your old key until you know that the new lock works.

What it really means: Keep the previous version ready to use again.

In testing: One command returns to the last good release.

export RELEASE=previous-image-version
docker compose up -d

Practise this before you need it. Database changes need extra care, so make them backward compatible, as you will see in Prompt Versioning and Evaluation Gates.


🌍 Real-Life Example: The Disk That Vanished

A tester deploys to a container platform with temporary storage. The agent works well until a routine restart wipes its database and every report. A named volume, or the right kind of persistent disk, would have kept them.


🧪 Try It Yourself

  1. Compare the options above for your team.
  2. Write down your server’s size and region.
  3. Draw your network, including what can reach what.
  4. If you have a cloud account, deploy the demo.
  5. Practise a rollback.

📋 Ready-to-Copy Prompt

You are a cloud engineer reviewing a first deployment of a QA agent service.

Plan:
[describe the hosting, network and storage with no secrets]

Check public exposure, persistence, outbound network limits, secrets,
updates and rollback. Suggest the cheapest setup that is still safe.

Tell me to confirm current details in my cloud provider's documentation.

⚠️ Common Mistakes

  • Starting with Kubernetes for one small service
  • Exposing the service directly to the internet
  • Using temporary storage for the database
  • Deploying without a rollback plan
  • Allowing unrestricted outbound network access

🎯 Key Takeaways

  • A small server is enough for a first release.
  • Data must live on persistent storage.
  • Keep the service private, and limit outbound traffic.
  • Practise your rollback before you need it.

🎤 Interview Questions

1. How would you deploy a browser-based AI agent?

Model answer: I would run a pinned container on a small server with persistent storage, private network access and secrets provided at runtime.

2. Why might serverless be a poor fit?

Model answer: Runs can be long, browsers need significant resources and local files may not persist. A worker on a server or container service suits it better.

3. How do you limit damage if the agent misbehaves?

Model answer: I restrict its outbound network access, give it narrow credentials, cap its resources and keep it away from production systems.

4. Why bind to 127.0.0.1 on the server?

Model answer: It keeps the application off the public interface. Access must then pass through a controlled proxy or VPN.

5. How do you roll back a bad deployment?

Model answer: I redeploy the previous pinned image. I keep database changes backward compatible, so the older version can still run.


📝 Assignment

Write a deployment plan covering hosting, network, storage, secrets and rollback. If you can, deploy the demo to a small server, then destroy it when you finish so that you are not charged for it.


❓ FAQ

Which cloud provider should I use?

Any major provider can run this design. Use the one your company has approved.

Can a team use my service without a VPN?

Only with strong protection, such as HTTPS and proper sign-in. A private network is the safer first choice.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Access Control: Secrets, Sign-In, Roles and Network Limits.

Playwright Docker Image: Package Your QA Agent

“It works on my machine” stops being a problem when your machine travels with the code. A Playwright Docker image packages your agent with the exact browsers and system libraries that it needs. In this lesson you will write a container image that is pinned, runs as a normal user and contains no secrets.

📖 What You’ll Learn

  • Why containers suit browser testing
  • How to pin the Playwright image
  • How to run as a non-root user
  • How to keep secrets out of an image

💡 Why Use the Playwright Docker Image?

Explain Like I’m 5: A Packed Lunchbox

Imagine: A lunchbox holds everything you need, so it does not matter which table you sit at.

What it really means: A container carries the app and everything that it depends on.

In testing: Browsers and system libraries always match your Playwright version.

The official image already includes the browsers. If you install Docker Desktop or Docker Engine, you can build and run it anywhere.

Match the versions exactly

The image tag must match the @playwright/test version in your package.json. If they differ, Playwright cannot find its browser files.


🔍 Write the Dockerfile

Explain Like I’m 5: Labelling the Jar

Imagine: A jar marked “jam” could hold anything. A jar marked “strawberry, June” tells you exactly what is inside.

What it really means: A pinned tag says exactly which version you are using.

In testing: Never use latest for a production image.

Create Dockerfile.

# Pin this to the same version as @playwright/test in package.json.
FROM mcr.microsoft.com/playwright:v1.65.0-noble

WORKDIR /app

COPY package.json package-lock.json ./
RUN npm ci

COPY . .
RUN npm run typecheck

# The app runs as a normal user that can write only to these folders.
RUN mkdir -p data artifacts reports playwright-report tests/generated \
  && chown -R pwuser:pwuser data artifacts reports playwright-report tests/generated

USER pwuser
ENV NODE_ENV=production
ENV SERVICE_HOST=0.0.0.0
EXPOSE 8080

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD node -e "fetch('http://127.0.0.1:8080/healthz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"

# Runs the demo target beside the service. Lesson 82 explains how to remove it.
CMD ["sh", "-c", "npx tsx demo/server.ts & exec npx tsx src/service/main.ts"]
LineWhy it is there
Pinned FROM tagThe same browsers on every build
COPY package*.json firstDependencies are cached between builds
RUN npm run typecheckA broken build cannot be shipped
USER pwuserThe app does not run as root
Limited chownThe app can write only to named folders
HEALTHCHECKThe platform can see when it is unhealthy

🧩 Keep Secrets Out

Explain Like I’m 5: Writing a Password on the Lunchbox

Imagine: Anyone who sees the box can read it, wherever it goes.

What it really means: Anything copied into an image travels with it.

In testing: The .env file is never included in the build.

Create .dockerignore.

node_modules
.git
.env
data
artifacts
reports
playwright-report
test-results
tests/generated

Secrets are given to the container when it starts, as you will see in Deploy an AI Agent to the Cloud. They should never be in the image, because anyone who can pull the image could read them.


🛠️ Build and Run It Locally

These commands need Docker to be installed. Build first, and read any errors carefully.

docker build -t qa-agent:local .

Run the container with the recommended options.

docker run --rm --init -p 127.0.0.1:8080:8080 --env-file .env -e AUTH_MODE=token qa-agent:local

The --init option helps to clean up browser child processes. Check it from another terminal.

curl http://127.0.0.1:8080/healthz

Your first build may reveal problems that a laptop hides, such as a file that was never committed. That is one of the benefits of building a container.


📌 Running Chromium Safely

The Playwright documentation advises against using this image to visit untrusted websites. That matches your project’s rule of testing only approved environments. For stronger browser isolation, the documentation describes running as pwuser together with a custom seccomp profile.

You can also review Playwright GitHub Actions to see the same browser setup in a pipeline.


🌍 Real-Life Example: The Missing Library

A tester’s agent runs perfectly on Windows. On a Linux server, Chromium fails with an error about a missing shared library. Installing system packages by hand becomes a long afternoon. The official image includes those libraries, so the problem never appears.


🧪 Try It Yourself

  1. Create both files.
  2. Confirm that the image tag matches your package.json.
  3. If you have Docker, build the image and run it.
  4. Run docker history and look for anything sensitive.
  5. Run whoami inside the container, and confirm that it is pwuser.

📋 Ready-to-Copy Prompt

You are a container security reviewer.

My Dockerfile:
[paste it]

My .dockerignore:
[paste it]

Check for: unpinned images, root users, secrets in layers, missing health
checks and folders that are writable when they should not be.

Return a table with: Line, Risk and Smallest Fix. Tell me to confirm the
current image tag on the official registry.

⚠️ Common Mistakes

  • Using the latest tag
  • Letting the image and package versions differ
  • Copying .env into the image
  • Running the service as root
  • Testing only on the developer’s machine

🎯 Key Takeaways

  • A container carries the browsers and libraries with it.
  • Pin the image to your exact Playwright version.
  • Run as a normal user with limited write access.
  • Pass secrets when the container starts.

🎤 Interview Questions

1. Why run Playwright in Docker?

Model answer: It provides the same browsers and system libraries everywhere, which avoids environment differences between machines and CI.

2. Why pin the Playwright image version?

Model answer: The tests and browser builds must match. A pinned tag also makes builds repeatable and protects against unexpected changes.

3. How do you keep secrets out of a Docker image?

Model answer: I exclude them with .dockerignore and provide them at runtime through the platform’s secret mechanism. Image layers can be read by anyone who pulls them.

4. Why not run as root in a container?

Model answer: If the app is compromised, a non-root user limits the damage. I also limit which folders the user can write to.

5. What does a health check do?

Model answer: It lets the platform detect an unhealthy container and restart or replace it.


📝 Assignment

Build your image and record its size. Add a second Dockerfile stage if you can make the image smaller safely. Then check it with a container scanner, and fix one finding.


❓ FAQ

Do I need Docker to complete this course?

No. It is needed only if you deploy with containers. Read the lesson and come back to it later if necessary.

Why does the image include the demo app?

It lets you prove the deployment without touching any other system. The next lesson shows how to point it at your own test environment.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Deploy an AI Agent to the Cloud: A Safe First Release.

Agent Workflow State: Save Runs, Traces and Evidence

When something goes wrong in production, your evidence is everything. Good agent workflow state answers what happened, in what order and who approved it. In this lesson you will find a real problem in the project so far, where each run destroys the previous run’s evidence, and fix it.

📖 What You’ll Learn

  • What to store for every run
  • Why each run needs its own evidence
  • How to retain and delete artifacts safely
  • What to do about backups

💡 What to Keep for Agent Workflow State

Explain Like I’m 5: A Ship’s Logbook

Imagine: The captain writes down every change of course. After a storm, anyone can see what happened.

What it really means: A run’s state is its logbook.

In testing: It records each stage, approval and result.

DataWhere it livesWhy
Run status and reasonruns tableAnswers “what happened?”
Full workflow stateruns.state_jsonExplains each stage
Approvalsapprovals tableShows who decided what
Actionsaudit_log tableProvides a permanent record
Reportsreports/ folderIs shared with readers
Traces and screenshotsartifacts/runs/Is the evidence for diagnosis

You began saving state in AI Agent Orchestrator. The service now also stores it in the database.


🔍 Find the Problem

Explain Like I’m 5: Writing on the Same Page

Imagine: If you write today’s homework on top of yesterday’s, you lose yesterday’s work.

What it really means: Each run must not overwrite the one before it.

In testing: Playwright reuses the same results folder for every run.

Look at playwright.config.ts. Results always go to artifacts/test-results and artifacts/results.json. On your laptop that was fine. In a service, tomorrow’s run deletes today’s evidence.


🧩 Archive Each Run

Create src/service/artifacts.ts.

import { cp, mkdir, readdir, rm, stat } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { resolve } from 'node:path';

const DAY_MS = 86_400_000;

// The next run overwrites these files, so each run keeps its own copy.
export async function archiveRunArtifacts(root: string, runId: string): Promise<void> {
  const target = resolve(root, 'artifacts/runs', runId, 'final');
  await mkdir(target, { recursive: true });
  for (const name of ['test-results', 'results.json']) {
    const source = resolve(root, 'artifacts', name);
    if (existsSync(source)) {
      await cp(source, resolve(target, name), { recursive: true });
    }
  }
}

export async function deleteExpiredArtifacts(
  root: string,
  maxAgeDays: number,
  now: number = Date.now(),
): Promise<string[]> {
  const folder = resolve(root, 'artifacts/runs');
  if (!existsSync(folder)) {
    return [];
  }
  const deleted: string[] = [];
  for (const name of await readdir(folder)) {
    const path = resolve(folder, name);
    if (now - (await stat(path)).mtimeMs > maxAgeDays * DAY_MS) {
      await rm(path, { recursive: true, force: true });
      deleted.push(name);
    }
  }
  return deleted;
}

Now call it when a run finishes. In src/service/main.ts, add the import, and add this option to the Worker.

onFinished: async (run) => {
  await archiveRunArtifacts(root, run.id);
},

The worker waits for this before it takes the next job, so the next run cannot overwrite the files first.

Create tests/unit/artifacts.test.ts.

import assert from 'node:assert/strict';
import { existsSync } from 'node:fs';
import { mkdir, mkdtemp, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { test } from 'node:test';
import { archiveRunArtifacts, deleteExpiredArtifacts } from '../../src/service/artifacts.js';

test('each run keeps its own copy of the evidence', async () => {
  const root = await mkdtemp(join(tmpdir(), 'qa-agent-'));
  await mkdir(join(root, 'artifacts/test-results'), { recursive: true });
  await writeFile(join(root, 'artifacts/test-results/trace.zip'), 'trace');
  await writeFile(join(root, 'artifacts/results.json'), '{}');
  await archiveRunArtifacts(root, 'run-1');
  assert.ok(existsSync(join(root, 'artifacts/runs/run-1/final/test-results/trace.zip')));
  assert.ok(existsSync(join(root, 'artifacts/runs/run-1/final/results.json')));
});

test('only old runs are deleted', async () => {
  const root = await mkdtemp(join(tmpdir(), 'qa-agent-'));
  await mkdir(join(root, 'artifacts/runs/run-1'), { recursive: true });
  assert.deepEqual(await deleteExpiredArtifacts(root, 30), []);
  const later = Date.now() + 31 * 86_400_000;
  assert.deepEqual(await deleteExpiredArtifacts(root, 30, later), ['run-1']);
  assert.equal(existsSync(join(root, 'artifacts/runs/run-1')), false);
});

🛠️ Decide How Long to Keep Things

Explain Like I’m 5: Clearing the Attic

Imagine: You keep important papers and throw out old junk. If you keep everything, you cannot find anything.

What it really means: Set a clear time limit for each kind of data.

In testing: Keep traces for 30 days and audit records for longer.

DataExample retentionReason
Traces and screenshots30 daysLarge, and may show page content
Reports90 daysUseful for comparing runs
Audit logAs long as policy requiresProvides accountability

These are only examples, so follow your organisation’s policy. Call deleteExpiredArtifacts from a scheduled job.

Evidence can contain sensitive data

Traces record page content and network traffic. This is one more reason to use synthetic data, and to restrict who can read artifacts. Review AI Data Privacy for Testers.


📌 Moving to Object Storage

Local folders are enough for one server. When you outgrow them, upload each run’s folder to object storage, such as S3, Google Cloud Storage or Azure Blob Storage. Keep the bucket private, and share files using short-lived links.


🔑 Backups

A backup you have never restored is only a hope. For SQLite, use a proper backup method and not a plain file copy of a live database, because recent changes may sit in a separate write-ahead log.

  • Schedule backups, and store them away from the server.
  • Restore one on a test server, regularly.
  • Write down how long a restore takes.

🌍 Real-Life Example: The Missing Trace

A release fails on Tuesday, and on Wednesday a developer asks for the Playwright trace. It has gone, because Tuesday night’s run replaced it. With one folder per run, the trace is still there, under that run’s ID.


🧪 Try It Yourself

  1. Add the archive step and run the unit tests.
  2. Complete two runs through the service.
  3. Confirm that each has its own final folder.
  4. Read a run’s state_json from the database.
  5. Decide on retention periods for your team.

📋 Ready-to-Copy Prompt

You are a senior engineer reviewing data retention for a QA agent service.

What we store: [list data types and where they live]
Our rules: [paste your retention policy with no confidential detail]

Find any data kept too long or not long enough, any place where sensitive
data could appear and any gap in our backup and restore plan.

Ask me about our legal requirements before suggesting retention periods.

⚠️ Common Mistakes

  • Letting each run overwrite the last
  • Keeping traces forever with no owner
  • Never testing a restore
  • Making artifact storage public
  • Copying a live database file as a backup

🎯 Key Takeaways

  • Every run needs a complete record.
  • Each run must keep its own evidence.
  • Set a retention period for each kind of data.
  • A backup only counts if you have restored it.

🎤 Interview Questions

1. What would you store for each agent run?

Model answer: I would store the status, stop reason, workflow state, approvals, audit events, reports, traces and screenshots, all tied to one run ID.

2. How do you stop artifacts from being overwritten?

Model answer: I archive them to a folder named after the run before the next run starts. The worker waits for that to finish.

3. Why are retention policies important?

Model answer: They control storage costs and reduce the risk from keeping sensitive data. Different data needs different lifetimes.

4. Why might a Playwright trace be sensitive?

Model answer: It can capture page content, network requests and form values. I use synthetic data and restrict who can read traces.

5. How do you know your backups work?

Model answer: I restore them regularly on a separate system and verify the data. An untested backup gives false confidence.


📝 Assignment

Add a scheduled call to deleteExpiredArtifacts, and log what it removes. Then write a one-page restore procedure, and test it on a copy of your data.


❓ FAQ

Can I store traces in the database?

It is better not to. They are large files, so keep them in storage and save only their location.

When should I use object storage?

Use it when you run more than one server or when local disk space becomes a limit.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Playwright Docker Image: Package Your QA Agent.

AI Agent as a Service: Wrap Your QA Agent in an API

Until now, a person has had to type yes in a terminal. An AI agent as a service works differently. A team member sends a request to an API, a background worker runs the job and an approver decides from wherever they are. You will reuse your orchestrator almost unchanged, because you kept it independent of how it is started.

📖 What You’ll Learn

  • How to design a small API for an agent
  • How approvals work without a terminal
  • How a worker runs jobs in the background
  • How to check that a service is healthy

💡 The API for an AI Agent as a Service

Explain Like I’m 5: A Post Box

Imagine: You drop a letter in the box and walk away. Later, you can ask whether it arrived.

What it really means: You submit a run and check on it later.

In testing: The API replies with a run ID straight away.

RequestPurposeWho may use it
POST /v1/runsStart a runRequester
GET /v1/runs/:idCheck its statusViewer
GET /v1/runs/:id/reportRead the reportViewer
GET /v1/approvalsSee pending approvalsApprover
POST /v1/approvals/:idApprove or rejectApprover
GET /healthz and /readyzCheck the serviceAnyone

Notice that a request names an environment, and it never gives a URL. People choose from approved environments, so nobody can point the agent at an arbitrary site.

Install the web framework.

npm install express
npm install -D @types/express

Create src/service/config.ts.

import 'dotenv/config';
import * as z from 'zod';

const ServiceSchema = z.object({
  SERVICE_HOST: z.string().default('127.0.0.1'),
  SERVICE_PORT: z.coerce.number().int().positive().default(8080),
  DATABASE_PATH: z.string().default('data/agent.db'),
  APPROVAL_TIMEOUT_MINUTES: z.coerce.number().positive().default(60),
  AUTH_MODE: z.enum(['local-dev', 'token']).default('token'),
  MAX_LLM_CALLS_PER_RUN: z.coerce.number().int().positive().default(20),
  MAX_TOKENS_PER_RUN: z.coerce.number().int().positive().default(200_000),
  MAX_DAILY_COST_USD: z.coerce.number().positive().default(10),
  INPUT_COST_PER_MILLION_USD: z.coerce.number().nonnegative().default(0),
  OUTPUT_COST_PER_MILLION_USD: z.coerce.number().nonnegative().default(0),
  RUNS_PER_HOUR_PER_USER: z.coerce.number().int().positive().default(10),
  SLACK_SIGNING_SECRET: z.string().default(''),
  SLACK_WEBHOOK_URL: z.string().default(''),
  SLACK_APPROVERS: z.string().default('{}'),
});

export type ServiceConfig = z.infer<typeof ServiceSchema>;

export function loadServiceConfig(): ServiceConfig {
  return ServiceSchema.parse(process.env);
}

Create src/service/security.ts. Roles are defined here, and Lesson 83 adds real sign-in.

import type { Request } from 'express';

export type Role = 'viewer' | 'requester' | 'approver' | 'admin';
export type Permission = 'runs:read' | 'runs:create' | 'approvals:decide' | 'ops:manage';
export type Actor = { id: string; role: Role };

const ROLE_PERMISSIONS: Record<Role, Permission[]> = {
  viewer: ['runs:read'],
  requester: ['runs:read', 'runs:create'],
  approver: ['runs:read', 'approvals:decide'],
  admin: ['runs:read', 'runs:create', 'approvals:decide', 'ops:manage'],
};

export function isAllowed(actor: Actor, permission: Permission): boolean {
  return ROLE_PERMISSIONS[actor.role].includes(permission);
}

export interface Security {
  authenticate(request: Request): Actor | null;
}

// For your own machine only. It trusts a header that anyone could set.
export class LocalDevSecurity implements Security {
  authenticate(request: Request): Actor | null {
    const id = request.header('x-dev-user');
    return id ? { id, role: 'admin' } : null;
  }
}

🔍 Approvals without a Terminal

Explain Like I’m 5: Leaving a Note

Imagine: You leave a note asking, “May I borrow the car?” Then you wait for a written answer. No answer means no.

What it really means: An approval is a database record that waits for a person.

In testing: If nobody answers in time, the run stops.

Your Approver interface from Validate AI-Generated Code pays off here. Create src/service/stored-approver.ts.

import { setTimeout as sleep } from 'node:timers/promises';
import type { ApprovalRequest, Approver } from '../core/approval.js';
import type { ApprovalRow, ServiceStore } from './store.js';

export type ApprovalNotifier = (approval: ApprovalRow) => Promise<void>;

export class StoredApprover implements Approver {
  constructor(
    private readonly store: ServiceStore,
    private readonly runId: string,
    private readonly timeoutMs: number,
    private readonly notify?: ApprovalNotifier,
    private readonly pollMs = 1000,
  ) {}

  async decide({ action, summary }: ApprovalRequest): Promise<boolean> {
    const expiresAt = Date.now() + this.timeoutMs;
    const approval = this.store.createApproval(this.runId, action, summary, expiresAt);
    await this.notify?.(approval).catch((error: unknown) => {
      this.store.addAudit('system', 'notification.failed', this.runId, String(error));
    });

    while (Date.now() < expiresAt) {
      const current = this.store.getApproval(approval.id);
      if (current && current.status !== 'pending') {
        return current.status === 'approved';
      }
      await sleep(this.pollMs);
    }
    // Nobody answered in time, so the action does not happen.
    this.store.decideApproval(approval.id, 'expired', 'system');
    return false;
  }

  close(): void {}
}

export type DecisionResult = 'ok' | 'not_found' | 'already_decided' | 'self_approval';

export function recordDecision(
  store: ServiceStore,
  approvalId: string,
  actorId: string,
  approved: boolean,
): DecisionResult {
  const approval = store.getApproval(approvalId);
  const run = approval ? store.getRun(approval.run_id) : undefined;
  if (!approval || !run) {
    return 'not_found';
  }
  if (run.requested_by === actorId) {
    return 'self_approval';
  }
  if (!store.decideApproval(approvalId, approved ? 'approved' : 'rejected', actorId)) {
    return 'already_decided';
  }
  store.addAudit(actorId, approved ? 'approval.approved' : 'approval.rejected', run.id, approval.action);
  return 'ok';
}

Two rules matter here. An unanswered request counts as a refusal, and a person cannot approve their own run.


🧩 Run the Workflow in a Worker

Create src/service/run-workflow.ts. It connects the agents from AI Agent Orchestrator.

import { ExplorerAgent } from '../agents/explorer.agent.js';
import { FailureAnalystAgent } from '../agents/failure-analyst.agent.js';
import { GeneratorAgent } from '../agents/generator.agent.js';
import { HealerAgent } from '../agents/healer.agent.js';
import { PlannerAgent } from '../agents/planner.agent.js';
import { ReporterAgent } from '../agents/reporter.agent.js';
import { RequirementParserAgent } from '../agents/requirement-parser.agent.js';
import type { Env } from '../config/env.js';
import { buildProjectContext } from '../config/project.js';
import type { Approver } from '../core/approval.js';
import { Orchestrator } from '../core/orchestrator.js';
import { createState } from '../core/workflow-state.js';
import type { LlmClient } from '../llm/llm-client.js';
import { createLlmClient } from '../llm/provider-adapter.js';
import { PlaywrightBrowserTool } from '../tools/browser.tool.js';
import { FileSystemTool } from '../tools/filesystem.tool.js';
import { TestRunnerTool } from '../tools/test-runner.tool.js';
import type { RunRow } from './store.js';

export type WorkflowOutcome = {
  status: 'completed' | 'stopped';
  stopReason?: string;
  reportPath: string;
  stateJson: string;
};

export type WorkflowRunner = (run: RunRow, approver: Approver) => Promise<WorkflowOutcome>;

export function createWorkflowRunner(
  root: string,
  env: Env,
  wrapLlm: (client: LlmClient, run: RunRow) => LlmClient = (client) => client,
): WorkflowRunner {
  const context = buildProjectContext(env);

  return async (run, approver) => {
    const llm = wrapLlm(createLlmClient(env), run);
    const orchestrator = new Orchestrator(
      root,
      context,
      {
        parser: new RequirementParserAgent(llm),
        explorer: new ExplorerAgent(new PlaywrightBrowserTool(true), context.allowedDomains),
        planner: new PlannerAgent(llm),
        generator: new GeneratorAgent(llm),
        analyst: new FailureAnalystAgent(llm),
        healer: new HealerAgent(llm),
        reporter: new ReporterAgent(root),
      },
      {
        files: new FileSystemTool(root),
        runner: new TestRunnerTool(root, env.ALLOW_TEST_EXECUTION === 'true'),
        approver,
      },
    );
    const state = createState(run.id, run.requirement, llm.providerName);
    const reportPath = await orchestrator.run(state);
    return {
      status: state.status === 'completed' ? 'completed' : 'stopped',
      stopReason: state.stopReason,
      reportPath,
      stateJson: JSON.stringify(state),
    };
  };
}

Create src/service/worker.ts.

import { setTimeout as sleep } from 'node:timers/promises';
import { StoredApprover, type ApprovalNotifier } from './stored-approver.js';
import type { WorkflowRunner } from './run-workflow.js';
import type { RunRow, ServiceStore } from './store.js';

export type WorkerOptions = {
  approvalTimeoutMs: number;
  notify?: ApprovalNotifier;
  onFinished?: (run: RunRow, status: string) => void | Promise<void>;
  leaseMs?: number;
  pollMs?: number;
};

export class Worker {
  private running = false;
  private readonly leaseMs: number;

  constructor(
    private readonly store: ServiceStore,
    private readonly runWorkflow: WorkflowRunner,
    private readonly options: WorkerOptions,
  ) {
    this.leaseMs = options.leaseMs ?? 60_000;
  }

  // Runs one queued job. Returns false when there was nothing to do.
  async tick(): Promise<boolean> {
    if (this.store.isPaused()) {
      return false;
    }
    const run = this.store.claimNextRun(this.leaseMs);
    if (!run) {
      return false;
    }

    const heartbeat = setInterval(
      () => this.store.renewLease(run.id, this.leaseMs),
      this.leaseMs / 3,
    );
    try {
      this.store.addAudit('worker', 'run.started', run.id);
      const approver = new StoredApprover(
        this.store,
        run.id,
        this.options.approvalTimeoutMs,
        this.options.notify,
        this.options.pollMs,
      );
      const outcome = await this.runWorkflow(run, approver);
      this.store.finishRun(run.id, outcome);
      this.store.addAudit('worker', `run.${outcome.status}`, run.id, outcome.stopReason);
      await this.options.onFinished?.(run, outcome.status);
    } catch (error) {
      const stopReason = error instanceof Error ? error.message : String(error);
      this.store.finishRun(run.id, { status: 'stopped', stopReason });
      this.store.addAudit('worker', 'run.failed', run.id, stopReason);
      await this.options.onFinished?.(run, 'stopped');
    } finally {
      clearInterval(heartbeat);
    }
    return true;
  }

  // One run at a time: runs share a generated tests folder and a results file.
  async start(): Promise<void> {
    this.running = true;
    this.store.interruptExpiredRuns();
    while (this.running) {
      const worked = await this.tick();
      if (!worked) {
        await sleep(this.options.pollMs ?? 1000);
      }
    }
  }

  stop(): void {
    this.running = false;
  }
}

The heartbeat renews the lease while a run is healthy. If the process dies, the lease expires and the run is marked as interrupted.


🛠️ Build the API

Explain Like I’m 5: The Front Desk

Imagine: The receptionist checks your name, takes your request and never goes into the back room.

What it really means: The API checks and records, and the worker does the work.

In testing: Invalid input is rejected before it reaches the queue.

Create src/service/api.ts.

import { readFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import express, {
  type Express,
  type NextFunction,
  type Request,
  type RequestHandler,
  type Response,
} from 'express';
import * as z from 'zod';
import { isAllowed, type Actor, type Permission, type Security } from './security.js';
import { recordDecision } from './stored-approver.js';
import type { ServiceStore } from './store.js';

const CreateRunSchema = z.object({
  requirement: z.string().min(1).max(20_000),
  environment: z.string().min(1),
});
const DecisionSchema = z.object({ decision: z.enum(['approve', 'reject']) });

export type Guard = (permission: Permission) => RequestHandler;

export type ApiOptions = {
  root: string;
  store: ServiceStore;
  security: Security;
  environments: string[];
  allowRun?: (actorId: string) => boolean;
  mount?: (app: Express, guard: Guard) => void;
};

const actorOf = (response: Response) => response.locals.actor as Actor;

export function createApi(options: ApiOptions): Express {
  const { root, store, security, environments } = options;
  const app = express();
  app.disable('x-powered-by');

  const guard: Guard = (permission) => (request, response, next) => {
    const actor = security.authenticate(request);
    if (!actor) {
      response.status(401).json({ error: 'Authentication required' });
    } else if (!isAllowed(actor, permission)) {
      response.status(403).json({ error: 'Not allowed' });
    } else {
      response.locals.actor = actor;
      next();
    }
  };

  options.mount?.(app, guard);
  app.use(express.json({ limit: '64kb' }));

  app.get('/healthz', (_request, response) => {
    response.json({ status: 'ok' });
  });

  app.get('/readyz', (_request, response) => {
    response.json({ status: store.ping() ? 'ready' : 'not_ready', paused: store.isPaused() });
  });

  app.post('/v1/runs', guard('runs:create'), (request, response) => {
    const body = CreateRunSchema.safeParse(request.body);
    if (!body.success) {
      response.status(400).json({ error: 'Invalid request body' });
      return;
    }
    if (!environments.includes(body.data.environment)) {
      response.status(400).json({ error: 'Environment is not approved' });
      return;
    }
    const actor = actorOf(response);
    if (options.allowRun && !options.allowRun(actor.id)) {
      response.status(429).json({ error: 'Too many runs. Try again later.' });
      return;
    }
    const run = store.createRun({
      requirement: body.data.requirement,
      environment: body.data.environment,
      requestedBy: actor.id,
      idempotencyKey: request.header('idempotency-key'),
    });
    store.addAudit(actor.id, 'run.requested', run.id);
    response.status(202).json({ id: run.id, status: run.status });
  });

  app.get('/v1/runs/:id', guard('runs:read'), (request, response) => {
    const run = store.getRun(String(request.params.id));
    if (!run) {
      response.status(404).json({ error: 'Run not found' });
      return;
    }
    response.json({
      id: run.id,
      status: run.status,
      requestedBy: run.requested_by,
      stopReason: run.stop_reason,
      reportPath: run.report_path,
    });
  });

  app.get('/v1/runs/:id/report', guard('runs:read'), async (request, response) => {
    const run = store.getRun(String(request.params.id));
    if (!run?.report_path) {
      response.status(404).json({ error: 'Report not found' });
      return;
    }
    response.type('text/markdown').send(await readFile(resolve(root, run.report_path), 'utf8'));
  });

  app.get('/v1/approvals', guard('approvals:decide'), (_request, response) => {
    response.json(store.listPendingApprovals());
  });

  app.post('/v1/approvals/:id', guard('approvals:decide'), (request, response) => {
    const body = DecisionSchema.safeParse(request.body);
    if (!body.success) {
      response.status(400).json({ error: 'Decision must be approve or reject' });
      return;
    }
    const result = recordDecision(
      store,
      String(request.params.id),
      actorOf(response).id,
      body.data.decision === 'approve',
    );
    const statusCodes = { ok: 200, not_found: 404, already_decided: 409, self_approval: 403 };
    response.status(statusCodes[result]).json({ result });
  });

  app.use((error: unknown, _request: Request, response: Response, next: NextFunction) => {
    if (response.headersSent) {
      next(error);
      return;
    }
    const badInput = error instanceof SyntaxError || (error as { status?: number }).status === 413;
    response.status(badInput ? 400 : 500).json({ error: badInput ? 'Invalid request' : 'Server error' });
  });

  return app;
}

📌 Start the Service

Create src/service/main.ts.

import { loadEnv } from '../config/env.js';
import { createApi } from './api.js';
import { loadServiceConfig } from './config.js';
import { createWorkflowRunner } from './run-workflow.js';
import { LocalDevSecurity } from './security.js';
import { ServiceStore } from './store.js';
import { Worker } from './worker.js';

// Temporary version. Lesson 88 adds real security, limits and monitoring.
const root = process.cwd();
const env = loadEnv();
const config = loadServiceConfig();
if (config.SERVICE_HOST !== '127.0.0.1') {
  throw new Error('This version may run only on 127.0.0.1');
}

const store = new ServiceStore(config.DATABASE_PATH);
const worker = new Worker(store, createWorkflowRunner(root, env), {
  approvalTimeoutMs: config.APPROVAL_TIMEOUT_MINUTES * 60_000,
});
const app = createApi({ root, store, security: new LocalDevSecurity(), environments: ['demo'] });

app.listen(config.SERVICE_PORT, config.SERVICE_HOST, () => {
  console.log(`Service running on port ${config.SERVICE_PORT}`);
});
void worker.start();

Local development only

LocalDevSecurity trusts a header that anyone could send. The check at the top stops this version from listening on a network.

Add the script, then start the demo and the service in separate terminals.

npm pkg set scripts.service="tsx src/service/main.ts"
npm run demo
npm run service

In a third terminal, request a run as Asha.

curl -X POST http://127.0.0.1:8080/v1/runs -H "Content-Type: application/json" -H "x-dev-user: asha" -d "{\"requirement\":\"A user can sign in.\",\"environment\":\"demo\"}"

List what is waiting, acting as Ben, and then approve each one. Do this twice, because there are two approvals.

curl http://127.0.0.1:8080/v1/approvals -H "x-dev-user: ben"
curl -X POST http://127.0.0.1:8080/v1/approvals/YOUR-APPROVAL-ID -H "Content-Type: application/json" -H "x-dev-user: ben" -d "{\"decision\":\"approve\"}"

Finally, check the run with GET /v1/runs/YOUR-RUN-ID. Try approving as asha, and you should get a 403 response.


🔑 Test the Worker

Create tests/unit/worker.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import type { WorkflowRunner } from '../../src/service/run-workflow.js';
import { ServiceStore } from '../../src/service/store.js';
import { recordDecision, StoredApprover } from '../../src/service/stored-approver.js';
import { Worker } from '../../src/service/worker.js';

const queue = (store: ServiceStore) =>
  store.createRun({ requirement: 'Login', environment: 'demo', requestedBy: 'asha' });

const finished: WorkflowRunner = async () => ({
  status: 'completed',
  reportPath: 'reports/x.md',
  stateJson: '{}',
});

test('the worker completes a queued run', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  await new Worker(store, finished, { approvalTimeoutMs: 1000 }).tick();
  assert.equal(store.getRun(run.id)?.status, 'completed');
});

test('a crash is recorded as a stopped run', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  const crash: WorkflowRunner = async () => {
    throw new Error('Browser crashed');
  };
  await new Worker(store, crash, { approvalTimeoutMs: 1000 }).tick();
  assert.equal(store.getRun(run.id)?.status, 'stopped');
  assert.equal(store.getRun(run.id)?.stop_reason, 'Browser crashed');
});

test('a paused service does not start new runs', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  store.setSetting('paused', 'true');
  await new Worker(store, finished, { approvalTimeoutMs: 1000 }).tick();
  assert.equal(store.getRun(run.id)?.status, 'queued');
});

test('an unanswered approval counts as a no', async () => {
  const store = new ServiceStore(':memory:');
  const approver = new StoredApprover(store, queue(store).id, 20, undefined, 5);
  assert.equal(await approver.decide({ action: 'run-generated-tests', summary: 'x' }), false);
});

test('a person cannot approve their own run', async () => {
  const store = new ServiceStore(':memory:');
  const run = queue(store);
  const approval = store.createApproval(run.id, 'x', 'y', Date.now() + 1000);
  assert.equal(recordDecision(store, approval.id, 'asha', true), 'self_approval');
  assert.equal(recordDecision(store, approval.id, 'ben', true), 'ok');
  assert.equal(recordDecision(store, 'apr-missing', 'ben', true), 'not_found');
});

🌍 Real-Life Example: Approve at Lunch

A QA lead is away from their desk when a run reaches its approval step. With a terminal prompt, the run would simply wait for one person at one machine. With stored approvals, a colleague can review it from another machine, and every decision is recorded.


🧪 Try It Yourself

  1. Create the files and run the unit tests.
  2. Run the service and complete one run.
  3. Try to approve your own run.
  4. Send an environment called production.
  5. Send broken JSON and read the response.

📋 Ready-to-Copy Prompt

You are a senior API engineer reviewing a service that wraps a QA agent.

Code:
[paste your routes with no secrets]

Check input validation, error messages, status codes and anything that lets
a caller reach systems outside the approved list.

Return a table with: Route, Problem and Smallest Fix.
Do not suggest a new framework.

⚠️ Common Mistakes

  • Letting callers provide a target URL
  • Treating a missing approval as a yes
  • Returning stack traces to callers
  • Allowing people to approve their own work
  • Exposing local-only security on a network

🎯 Key Takeaways

  • The API accepts work, and the worker runs it.
  • Approvals become stored records that wait.
  • Unanswered approvals count as a no.
  • The orchestrator did not need to change.

🎤 Interview Questions

1. How do you turn a CLI agent into a service?

Model answer: I add an API to accept requests, a queue and worker for the slow work and a database for state. I keep the core workflow independent of how it starts.

2. How do approvals work in a service?

Model answer: The workflow creates an approval record and waits. A person decides through the API, and the workflow continues. If it times out, the answer is no.

3. Why not let users pass a target URL?

Model answer: It could point the agent at unapproved or internal systems. I accept the name of a configured environment and map it to a known target.

4. What is the difference between health and readiness?

Model answer: Health shows that the process is alive. Readiness shows that it can do work, such as reaching its database and not being paused.

5. What is separation of duties?

Model answer: The person requesting an action cannot approve it. This reduces the risk of mistakes and misuse.


📝 Assignment

Add a GET /v1/runs route that returns a requester’s last ten runs. Protect it with runs:read, validate its query and add a unit test.


❓ FAQ

Why does the worker run in the same process as the API?

It keeps the first release simple. The two talk only through the database, so they can be separated later.

How do I send requests from Windows?

The commands work in Windows Terminal with curl.exe. A tool such as Postman also works well.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Agent Workflow State: Save Runs, Traces and Evidence.

AI Agent Architecture for Production: API, Queue, Workers and Storage

A browser run can take minutes, so a request cannot simply wait for it to finish. A production AI agent architecture splits the work: one part accepts requests, and another does the slow work in the background. In this lesson you will learn each part, then build the database that also acts as your job queue.

📖 What You’ll Learn

  • What each production component does
  • Why a queue separates requests from work
  • Why to start with a simple database
  • How to stop two workers claiming one job

💡 The Production AI Agent Architecture

Explain Like I’m 5: Ordering at a Café

Imagine: You order at the counter, get a number and sit down. The kitchen works through the orders, and you are called when yours is ready.

What it really means: The API takes the order, and a worker does the work.

In testing: The API replies at once, while tests run in the background.

Team member or tool
       |
       v
      API  ---> Database (runs, approvals, audit log)
                    ^
                    |
                 Worker ---> Orchestrator ---> Browser and tests
                    |
                    v
          Artifact storage (traces, screenshots, reports)
PartJob
APIAccepts requests and shows status
Job queueHolds work that is waiting
WorkerRuns one job at a time
DatabaseStores runs, approvals and the audit log
Artifact storageKeeps traces, screenshots and reports

Your AI agent orchestrator runs inside the worker, and it needs no changes.


🔍 Start Small

Explain Like I’m 5: A Shed Before a Factory

Imagine: You test your idea in a shed. You build a factory only when people are queuing outside.

What it really means: Start with the simplest setup that does the job.

In testing: One service with one database is enough for one team.

StageDatabase and queueWhen it fits
First releaseSQLite on one serverOne team, one worker
GrowingPostgreSQL with a job tableSeveral servers or workers
LargeA managed queue and object storageMany teams

We use SQLite, which is built into Node.js 22 and later. It needs no extra server. Its limit is clear: it suits a single server. Because your code uses a small store class, you can replace it later without rewriting the API or worker.

One run at a time is deliberate

Every run shares the generated tests folder and the results file. A single worker avoids runs overwriting each other. Running several at once would need a separate working folder for each run.


🧩 Build the Store

Explain Like I’m 5: The Last Biscuit

Imagine: Two children reach for the last biscuit. Only one hand can actually pick it up.

What it really means: The database lets only one worker claim each job.

In testing: A single SQL statement makes the claim atomic.

Create src/service/store.ts.

import { randomUUID } from 'node:crypto';
import { mkdirSync } from 'node:fs';
import { dirname } from 'node:path';
import { DatabaseSync } from 'node:sqlite';

export type RunStatus = 'queued' | 'running' | 'completed' | 'stopped' | 'interrupted';
export type ApprovalStatus = 'pending' | 'approved' | 'rejected' | 'expired';

export type RunRow = {
  id: string;
  status: RunStatus;
  requirement: string;
  environment: string;
  requested_by: string;
  stop_reason: string | null;
  report_path: string | null;
  state_json: string | null;
  lease_until: number | null;
  created_at: number;
};

export type ApprovalRow = {
  id: string;
  run_id: string;
  action: string;
  summary: string;
  status: ApprovalStatus;
  decided_by: string | null;
  expires_at: number;
};

export type TokenRow = { id: string; name: string; role: string; revoked_at: number | null };

const SCHEMA = `
CREATE TABLE IF NOT EXISTS runs (
  id TEXT PRIMARY KEY, status TEXT NOT NULL, requirement TEXT NOT NULL,
  environment TEXT NOT NULL, requested_by TEXT NOT NULL, idempotency_key TEXT UNIQUE,
  stop_reason TEXT, report_path TEXT, state_json TEXT, lease_until INTEGER,
  created_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS approvals (
  id TEXT PRIMARY KEY, run_id TEXT NOT NULL, action TEXT NOT NULL, summary TEXT NOT NULL,
  status TEXT NOT NULL, decided_by TEXT, decided_at INTEGER, expires_at INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS audit_log (
  id INTEGER PRIMARY KEY AUTOINCREMENT, at INTEGER NOT NULL, actor TEXT NOT NULL,
  action TEXT NOT NULL, run_id TEXT, detail TEXT
);
CREATE TABLE IF NOT EXISTS api_tokens (
  id TEXT PRIMARY KEY, name TEXT NOT NULL, role TEXT NOT NULL,
  token_hash TEXT NOT NULL UNIQUE, revoked_at INTEGER
);
CREATE TABLE IF NOT EXISTS settings (key TEXT PRIMARY KEY, value TEXT NOT NULL);
`;

export class ServiceStore {
  private readonly db: DatabaseSync;

  constructor(path: string) {
    if (path !== ':memory:') {
      mkdirSync(dirname(path), { recursive: true });
    }
    this.db = new DatabaseSync(path);
    this.db.exec('PRAGMA journal_mode = WAL; PRAGMA busy_timeout = 5000;');
    this.db.exec(SCHEMA);
  }

  ping(): boolean {
    return this.db.prepare('SELECT 1 AS ok').get() !== undefined;
  }

  createRun(input: {
    requirement: string;
    environment: string;
    requestedBy: string;
    idempotencyKey?: string;
  }): RunRow {
    const key = input.idempotencyKey ? `${input.requestedBy}:${input.idempotencyKey}` : null;
    this.db
      .prepare(
        `INSERT INTO runs (id, status, requirement, environment, requested_by, idempotency_key, created_at)
         VALUES (?, 'queued', ?, ?, ?, ?, ?) ON CONFLICT(idempotency_key) DO NOTHING`,
      )
      .run(`run-${randomUUID()}`, input.requirement, input.environment, input.requestedBy, key, Date.now());
    const row = key
      ? this.db.prepare('SELECT * FROM runs WHERE idempotency_key = ?').get(key)
      : this.db.prepare('SELECT * FROM runs ORDER BY rowid DESC LIMIT 1').get();
    return row as unknown as RunRow;
  }

  getRun(id: string): RunRow | undefined {
    return this.db.prepare('SELECT * FROM runs WHERE id = ?').get(id) as unknown as RunRow | undefined;
  }

  // One SQL statement, so two workers can never claim the same run.
  claimNextRun(leaseMs: number): RunRow | undefined {
    return this.db
      .prepare(
        `UPDATE runs SET status = 'running', lease_until = ?
         WHERE id = (SELECT id FROM runs WHERE status = 'queued' ORDER BY created_at, rowid LIMIT 1)
         RETURNING *`,
      )
      .get(Date.now() + leaseMs) as unknown as RunRow | undefined;
  }

  renewLease(id: string, leaseMs: number): void {
    this.db.prepare('UPDATE runs SET lease_until = ? WHERE id = ?').run(Date.now() + leaseMs, id);
  }

  finishRun(id: string, result: {
    status: RunStatus;
    stopReason?: string;
    reportPath?: string;
    stateJson?: string;
  }): void {
    this.db
      .prepare(
        `UPDATE runs SET status = ?, stop_reason = ?, report_path = ?, state_json = ?, lease_until = NULL
         WHERE id = ?`,
      )
      .run(result.status, result.stopReason ?? null, result.reportPath ?? null, result.stateJson ?? null, id);
  }

  // A crashed run may already have had side effects, so it is never retried automatically.
  interruptExpiredRuns(): number {
    const result = this.db
      .prepare(
        `UPDATE runs SET status = 'interrupted', stop_reason = 'The worker stopped during this run'
         WHERE status = 'running' AND lease_until < ?`,
      )
      .run(Date.now());
    return Number(result.changes);
  }

  createApproval(runId: string, action: string, summary: string, expiresAt: number): ApprovalRow {
    const id = `apr-${randomUUID()}`;
    this.db
      .prepare(
        `INSERT INTO approvals (id, run_id, action, summary, status, expires_at)
         VALUES (?, ?, ?, ?, 'pending', ?)`,
      )
      .run(id, runId, action, summary, expiresAt);
    return this.getApproval(id) as ApprovalRow;
  }

  getApproval(id: string): ApprovalRow | undefined {
    return this.db.prepare('SELECT * FROM approvals WHERE id = ?').get(id) as unknown as
      | ApprovalRow
      | undefined;
  }

  listPendingApprovals(): ApprovalRow[] {
    return this.db
      .prepare(`SELECT * FROM approvals WHERE status = 'pending' ORDER BY expires_at`)
      .all() as unknown as ApprovalRow[];
  }

  // Only a pending approval can change, so the first decision always wins.
  decideApproval(id: string, status: ApprovalStatus, decidedBy: string): boolean {
    const result = this.db
      .prepare(
        `UPDATE approvals SET status = ?, decided_by = ?, decided_at = ?
         WHERE id = ? AND status = 'pending'`,
      )
      .run(status, decidedBy, Date.now(), id);
    return result.changes === 1;
  }

  addAudit(actor: string, action: string, runId?: string, detail?: string): void {
    this.db
      .prepare('INSERT INTO audit_log (at, actor, action, run_id, detail) VALUES (?, ?, ?, ?, ?)')
      .run(Date.now(), actor, action, runId ?? null, detail ?? null);
  }

  listAudit(runId: string): Array<{ actor: string; action: string }> {
    return this.db
      .prepare('SELECT actor, action FROM audit_log WHERE run_id = ? ORDER BY id')
      .all(runId) as unknown as Array<{ actor: string; action: string }>;
  }

  addToken(name: string, role: string, tokenHash: string): void {
    this.db
      .prepare('INSERT INTO api_tokens (id, name, role, token_hash) VALUES (?, ?, ?, ?)')
      .run(randomUUID(), name, role, tokenHash);
  }

  findToken(tokenHash: string): TokenRow | undefined {
    return this.db
      .prepare('SELECT id, name, role, revoked_at FROM api_tokens WHERE token_hash = ?')
      .get(tokenHash) as unknown as TokenRow | undefined;
  }

  revokeToken(name: string): void {
    this.db.prepare('UPDATE api_tokens SET revoked_at = ? WHERE name = ?').run(Date.now(), name);
  }

  getSetting(key: string): string | undefined {
    const row = this.db.prepare('SELECT value FROM settings WHERE key = ?').get(key) as
      | { value: string }
      | undefined;
    return row?.value;
  }

  setSetting(key: string, value: string): void {
    this.db
      .prepare(
        `INSERT INTO settings (key, value) VALUES (?, ?)
         ON CONFLICT(key) DO UPDATE SET value = excluded.value`,
      )
      .run(key, value);
  }

  isPaused(): boolean {
    return this.getSetting('paused') === 'true';
  }

  close(): void {
    this.db.close();
  }
}
DesignWhat it prevents
claimNextRun is one statementTwo workers taking one job
idempotency_key is uniqueA retried request creating a second run
Crashed runs become interruptedRepeating side effects automatically
decideApproval needs a pending rowA later decision overriding the first
audit_log tableLosing track of who did what

🛠️ Test the Store

Create tests/unit/store.test.ts.

import assert from 'node:assert/strict';
import { test } from 'node:test';
import { ServiceStore } from '../../src/service/store.js';

const newRun = (store: ServiceStore, key?: string) =>
  store.createRun({ requirement: 'Login', environment: 'demo', requestedBy: 'asha', idempotencyKey: key });

test('a queued run can be claimed only once', () => {
  const store = new ServiceStore(':memory:');
  newRun(store);
  assert.equal(store.claimNextRun(60_000)?.status, 'running');
  assert.equal(store.claimNextRun(60_000), undefined);
});

test('the same idempotency key does not create a second run', () => {
  const store = new ServiceStore(':memory:');
  assert.equal(newRun(store, 'abc').id, newRun(store, 'abc').id);
});

test('a run with an expired lease is interrupted, not retried', () => {
  const store = new ServiceStore(':memory:');
  const run = newRun(store);
  store.claimNextRun(-1);
  assert.equal(store.interruptExpiredRuns(), 1);
  assert.equal(store.getRun(run.id)?.status, 'interrupted');
  assert.equal(store.claimNextRun(60_000), undefined);
});

test('the first decision on an approval wins', () => {
  const store = new ServiceStore(':memory:');
  const approval = store.createApproval(newRun(store).id, 'x', 'y', Date.now() + 1000);
  assert.equal(store.decideApproval(approval.id, 'rejected', 'ben'), true);
  assert.equal(store.decideApproval(approval.id, 'approved', 'cara'), false);
  assert.equal(store.getApproval(approval.id)?.status, 'rejected');
});

Run the tests with npm run test:unit.


🌍 Real-Life Example: The Double Run

A tester’s network drops just after they send a request, so their tool retries. Without protection, two identical runs start and file two sets of bugs. With an idempotency key, the second request simply gets the first run back. You learned why retries are risky in MCP Server Testing.


🧪 Try It Yourself

  1. Create the store and run its tests.
  2. Open a store with a file path, and create a run.
  3. Stop your program and read the run again.
  4. Add a test that claims two jobs in order.
  5. Explain why crashed runs are not retried.

📋 Ready-to-Copy Prompt

You are a senior backend engineer reviewing a job queue built on one
database table.

Code:
[paste your store with no secrets]

Check whether two workers could ever claim the same job, whether a retry
could create a duplicate and what happens if a worker crashes.

Tell me the exact point at which this design would need PostgreSQL or a
managed queue. Do not suggest a change for a problem I do not have yet.

⚠️ Common Mistakes

  • Making the caller wait for a browser run
  • Adding several services before they are needed
  • Retrying interrupted runs automatically
  • Letting two runs share one output folder
  • Skipping the audit log

🎯 Key Takeaways

  • The API accepts work, and the worker does it.
  • Start with one simple database.
  • Claiming a job must be atomic.
  • Keep a clear path for growing later.

🎤 Interview Questions

1. Why use a job queue for an AI agent?

Model answer: Runs are slow, so the API should accept a request and respond quickly. A queue holds the work, controls concurrency and survives restarts.

2. How do you stop two workers processing the same job?

Model answer: I claim it with one atomic update in the database. Only one worker can change a queued job to running.

3. What is an idempotency key?

Model answer: It is a value that identifies a request. If the same key arrives again, the system returns the original result and creates nothing new.

4. Why not retry a crashed run automatically?

Model answer: It may already have had side effects. I mark it as interrupted and let a person decide whether to run it again.

5. When would you move away from SQLite?

Model answer: I would move when I need more than one server, more concurrent writes or managed backups. The store interface makes that change contained.


📝 Assignment

Add a listRecentRuns method with a limit, and write a unit test for it. Then sketch how the store’s methods would map to PostgreSQL tables and identify one that would need to change.


❓ FAQ

Is SQLite really acceptable in production?

For one small service on one server, it can be a sensible start. Back it up, and know when to move on.

Do I need Redis or Kafka?

Not for this workload. A team’s handful of test runs fits easily in a database table.


🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI Agent as a Service: Wrap Your QA Agent in an API.