AI Agent Production Readiness: Rollout and Incident Runbook

Launching to everyone on day one is how small problems become big ones. AI agent production readiness means proving that the system is safe to operate, then releasing it to a few people at a time. In this lesson you will add an emergency pause switch, finish the service and write the runbook you hope never to need.

---

💡 The AI Agent Production Readiness Review

Area	Evidence you need
Safety	Allowlists tested, and approvals required
Security	Token tests pass, and secrets are not in the image
Reliability	Crash and restart behaviour tested
Data	A backup has been restored
Cost	Budgets tested, and a provider cap is set
Monitoring	Alerts fire, and someone receives them
Quality	Evaluation gate passes
Operations	Owner named, and the runbook has been rehearsed

You started this kind of checking with AI Agent Acceptance Criteria. This review extends it to a live service.

---

🔍 Add the Emergency Pause

Create src/service/ops-routes.ts.

import express, { type Express } from 'express';
import * as z from 'zod';
import { log } from '../observability/logger.js';
import type { Metrics } from '../observability/metrics.js';
import type { Guard } from './api.js';
import type { Actor } from './security.js';
import type { ServiceStore } from './store.js';

const PauseSchema = z.object({ paused: z.boolean(), reason: z.string().min(1).max(500) });

export function mountOpsRoutes(
  app: Express,
  guard: Guard,
  store: ServiceStore,
  metrics: Metrics,
): void {
  app.get('/metrics', guard('ops:manage'), (_request, response) => {
    response.type('text/plain').send(metrics.render());
  });

  app.post('/v1/ops/pause', guard('ops:manage'), express.json(), (request, response) => {
    const body = PauseSchema.safeParse(request.body);
    if (!body.success) {
      response.status(400).json({ error: 'paused and reason are required' });
      return;
    }
    const actor = response.locals.actor as Actor;
    store.setSetting('paused', String(body.data.paused));
    store.addAudit(actor.id, body.data.paused ? 'ops.paused' : 'ops.resumed', undefined, body.data.reason);
    log('warn', 'ops.pause_changed', { actor: actor.id, paused: body.data.paused });
    response.json({ paused: body.data.paused });
  });
}

The Worker already checks isPaused before it takes a job. A paused service still answers status requests, so people can see what is going on. To pause it:

curl -X POST http://127.0.0.1:8080/v1/ops/pause -H "Authorization: Bearer YOUR-ADMIN-TOKEN" -H "Content-Type: application/json" -d "{\"paused\":true,\"reason\":\"Investigating bad results\"}"

---

🧩 Finish the Service

Replace src/service/main.ts with the final version, which brings together everything from this level, including AI Agent Access Control.

import { randomUUID } from 'node:crypto';
import { loadEnv } from '../config/env.js';
import { createSlackNotifier, mountSlackRoutes } from '../integrations/slack.js';
import { GuardedLlmClient } from '../llm/guarded-llm.js';
import { usageMeter } from '../llm/usage.js';
import { log } from '../observability/logger.js';
import { Metrics } from '../observability/metrics.js';
import { createApi } from './api.js';
import { archiveRunArtifacts } from './artifacts.js';
import { loadServiceConfig } from './config.js';
import { FixedWindowLimiter, StoredSpendLedger } from './limits.js';
import { mountOpsRoutes } from './ops-routes.js';
import { createWorkflowRunner } from './run-workflow.js';
import { LocalDevSecurity } from './security.js';
import { ServiceStore } from './store.js';
import { TokenSecurity } from './token-security.js';
import { Worker } from './worker.js';

const root = process.cwd();
const env = loadEnv();
const config = loadServiceConfig();
const store = new ServiceStore(config.DATABASE_PATH);
const metrics = new Metrics();
const ledger = new StoredSpendLedger(store);
const limiter = new FixedWindowLimiter(config.RUNS_PER_HOUR_PER_USER, 3_600_000);

if (config.AUTH_MODE === 'local-dev' && config.SERVICE_HOST !== '127.0.0.1') {
  throw new Error('AUTH_MODE=local-dev is allowed only on 127.0.0.1');
}

const runWorkflow = createWorkflowRunner(root, env, (client, run) => {
  usageMeter.reset();
  return new GuardedLlmClient(
    client,
    usageMeter,
    {
      maxCallsPerRun: config.MAX_LLM_CALLS_PER_RUN,
      maxTokensPerRun: config.MAX_TOKENS_PER_RUN,
      maxDailyCostUsd: config.MAX_DAILY_COST_USD,
      inputCostPerMillionUsd: config.INPUT_COST_PER_MILLION_USD,
      outputCostPerMillionUsd: config.OUTPUT_COST_PER_MILLION_USD,
    },
    ledger,
    metrics,
    run.id,
  );
});

const worker = new Worker(store, runWorkflow, {
  approvalTimeoutMs: config.APPROVAL_TIMEOUT_MINUTES * 60_000,
  notify: config.SLACK_WEBHOOK_URL ? createSlackNotifier(config.SLACK_WEBHOOK_URL) : undefined,
  onFinished: async (run, status) => {
    await archiveRunArtifacts(root, run.id);
    metrics.increment('qa_agent_runs_total', { status });
    log('info', 'run.finished', { runId: run.id, status });
  },
});

const app = createApi({
  root,
  store,
  security: config.AUTH_MODE === 'token' ? new TokenSecurity(store) : new LocalDevSecurity(),
  environments: ['demo'],
  allowRun: (actorId) => limiter.allow(actorId),
  mount: (server, guard) => {
    mountOpsRoutes(server, guard, store, metrics);
    if (config.SLACK_SIGNING_SECRET) {
      const approvers = JSON.parse(config.SLACK_APPROVERS) as Record<string, string>;
      mountSlackRoutes(server, store, config.SLACK_SIGNING_SECRET, approvers);
    }
  },
});

const server = app.listen(config.SERVICE_PORT, config.SERVICE_HOST, () => {
  log('info', 'service.started', { instance: randomUUID(), port: config.SERVICE_PORT });
});

void worker.start();

for (const signal of ['SIGTERM', 'SIGINT'] as const) {
  process.on(signal, () => {
    log('warn', 'service.stopping', { signal });
    worker.stop();
    server.close(() => process.exit(0));
  });
}

Notice the check near the top. The service refuses to start with local security on anything but your own machine. The final section shuts down gracefully when the platform asks it to stop.

---

🛠️ Roll Out in Stages

Stage	Who	Move on when
Shadow	The owner only, and results are not used	Runs complete reliably
Pilot	One team, one workflow	People accept its output
Limited	A few teams	Cost and support are manageable
General	Everyone approved	Service goals have been met

Write down what will make you stop as well as what will make you continue.

---

📌 Write the Incident Runbook

A runbook tells a tired person exactly what to do.

Step	Action
1. Stop the harm	Pause the service
2. Tell people	Post in the agreed channel
3. Save the evidence	Do not delete artifacts or logs
4. Find the scope	Which runs, systems and data were affected?
5. Fix or roll back	Redeploy the last good release
6. Rotate secrets	If any exposure is possible
7. Resume carefully	Unpause and watch the first runs
8. Review	Write a blameless review and add a test

Symptom	First check
Runs stuck in queued	Is the service paused? Is the worker alive?
Runs interrupted	Did the service restart or run out of memory?
Many model errors	Provider status, key and budget
Browser crashes	Memory and shared memory limits
Unexpected site visited	Pause, and treat it as a security incident

---

🌍 Real-Life Example: The Friday Launch

A team opens an agent to the whole company on a Friday afternoon. A prompt problem floods Jira with duplicate bugs, and the only person who knows how to stop it has gone home. A pilot group, a pause switch and a runbook would have made that a small and boring event.

---

🧪 Try It Yourself

1. Create the files, and start the final service.
2. Pause it, queue a run and confirm that it waits.
3. Resume it and watch the run start.
4. Fill in the readiness table with real evidence.
5. Rehearse the runbook with a colleague.

---

📋 Ready-to-Copy Prompt

You are running a production readiness review for a QA agent service.

Evidence for each area:
[paste your completed review with no secrets]

For each area, say whether the evidence is enough. List what could still
cause an incident on launch day, and ask me the five hardest questions.

Do not accept anything marked as done unless the evidence supports it.

---

⚠️ Common Mistakes

• Launching to everyone at once
• Having no way to stop the agent quickly
• Writing a runbook that nobody has rehearsed
• Deleting evidence during an incident
• Launching just before a weekend

---

🎯 Key Takeaways

• Readiness is proven with evidence.
• Roll out to small groups first.
• A pause switch must work without a deployment.
• Rehearse your incident response.

---

🎤 Interview Questions

1. How would you roll out an AI agent?

Model answer: I would go from shadow mode to a pilot, then limited and general release. Each stage has written criteria for moving on or stopping.

2. What is a kill switch?

Model answer: It is a fast way to stop the system without deploying. Here, an admin pauses the worker, so no new runs start.

3. What is the first step in an incident?

Model answer: Stop further harm, then communicate and preserve evidence. Understanding the cause comes after containment.

4. What should a production readiness review cover?

Model answer: It should cover safety, security, reliability, data, cost, monitoring, quality and operations, each with evidence.

5. What is a blameless review?

Model answer: It focuses on how the system allowed a failure, and not on who to blame. People speak openly, and the system improves.

---

📝 Assignment

Complete the readiness review for your service, and write a one-page runbook. Run a practice incident in which the agent visits an unapproved site, and record what you would improve.

---

❓ FAQ

Does a pause stop a run already in progress?

No. It stops new runs. To stop one immediately, stop the service, and the run will be marked as interrupted.

How long should a pilot last?

Long enough to see real usage and at least one failure. For many teams, that is a few weeks.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Measure AI Testing ROI: Jira and GitHub Integration for QA Agents.