AI Agent Orchestrator: Run the Whole Workflow with a Mock LLM

This is the exciting moment when the separate parts become one system. An AI agent orchestrator runs every stage in order, asks for approvals and stops safely when something goes wrong. You will finish this lesson with an agent that generates and runs real Playwright tests, using no paid model at all.

---

💡 How the AI Agent Orchestrator Works

parse --> explore --> plan --> generate --> validate
  --> write (approval) --> static checks
  --> execute (approval) --> analyse --> heal --> report

You designed a small version of this in Multi-Agent Orchestration for QA.

---

🔍 Save the Run State

Create src/core/workflow-state.ts.

import { mkdir, writeFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import type { ParsedRequirement } from '../schemas/requirement.schema.js';
import type { ScenarioPlan } from '../schemas/scenario.schema.js';
import type { GeneratedFile } from '../schemas/generation.schema.js';
import type { FailureAnalysis, HealingProposal } from '../schemas/failure.schema.js';
import type { ApprovalRecord } from './approval.js';
import type { PageObservation, TestRunResult } from './types.js';

export type WorkflowState = {
  runId: string;
  status: 'running' | 'completed' | 'stopped';
  stopReason?: string;
  completedStages: string[];
  requirementText: string;
  llmProvider: string;
  parsed?: ParsedRequirement;
  observation?: PageObservation;
  plan?: ScenarioPlan;
  generated?: GeneratedFile;
  firstRun?: TestRunResult;
  finalRun?: TestRunResult;
  analyses: Array<FailureAnalysis & { testTitle: string }>;
  healing?: HealingProposal & { applied: boolean };
  approvals: ApprovalRecord[];
};

export function createState(runId: string, requirementText: string, llmProvider: string): WorkflowState {
  return {
    runId,
    status: 'running',
    completedStages: [],
    requirementText,
    llmProvider,
    analyses: [],
    approvals: [],
  };
}

export async function saveState(root: string, state: WorkflowState): Promise<void> {
  const folder = resolve(root, 'artifacts/runs', state.runId);
  await mkdir(folder, { recursive: true });
  await writeFile(resolve(folder, 'state.json'), JSON.stringify(state, null, 2), 'utf8');
}

Create src/schemas/failure.schema.ts. Two of the temporary agents below need it.

import * as z from 'zod';

export const FailureAnalysisSchema = z.object({
  category: z.enum([
    'product_defect',
    'test_defect',
    'environment',
    'test_data',
    'flakiness',
    'unknown',
  ]),
  summary: z.string().min(1),
  evidence: z.array(z.string()),
  confidence: z.enum(['low', 'medium', 'high']),
  alternatives: z.array(z.string()),
});

export const HealingProposalSchema = z.object({
  path: z.string().min(1),
  oldText: z.string().min(1),
  newText: z.string().min(1),
  explanation: z.string().min(1),
});

export type FailureAnalysis = z.infer<typeof FailureAnalysisSchema>;
export type HealingProposal = z.infer<typeof HealingProposalSchema>;

---

🧩 Build the Orchestrator

Create src/core/orchestrator.ts.

import { stripVTControlCharacters } from 'node:util';
import type { FailureAnalystAgent } from '../agents/failure-analyst.agent.js';
import type { GeneratorAgent } from '../agents/generator.agent.js';
import type { ExplorerAgent } from '../agents/explorer.agent.js';
import type { HealerAgent } from '../agents/healer.agent.js';
import type { PlannerAgent } from '../agents/planner.agent.js';
import type { ReporterAgent } from '../agents/reporter.agent.js';
import type { RequirementParserAgent } from '../agents/requirement-parser.agent.js';
import type { FileSystemTool } from '../tools/filesystem.tool.js';
import type { TestRunnerTool } from '../tools/test-runner.tool.js';
import { validateGeneratedFile } from '../validators/generated-code.validator.js';
import type { ApprovalAction, Approver } from './approval.js';
import { LIMITS } from './guardrails.js';
import type { ProjectContext } from './types.js';
import { saveState, type WorkflowState } from './workflow-state.js';

export type Team = {
  parser: RequirementParserAgent;
  explorer: ExplorerAgent;
  planner: PlannerAgent;
  generator: GeneratorAgent;
  analyst: FailureAnalystAgent;
  healer: HealerAgent;
  reporter: ReporterAgent;
};

export type Runner = Pick<TestRunnerTool, 'run' | 'runGeneratedTests' | 'preserveArtifacts'>;

export type Tools = { files: FileSystemTool; runner: Runner; approver: Approver };

class StopWorkflow extends Error {}

export class Orchestrator {
  constructor(
    private readonly root: string,
    private readonly context: ProjectContext,
    private readonly team: Team,
    private readonly tools: Tools,
  ) {}

  private async approve(state: WorkflowState, action: ApprovalAction, summary: string) {
    const approved = await this.tools.approver.decide({ action, summary });
    state.approvals.push({ action, summary, approved, decidedAt: new Date().toISOString() });
    return approved;
  }

  private stages(state: WorkflowState): Array<[string, () => Promise<void>]> {
    const { team, tools, context } = this;
    return [
      ['parse', async () => {
        state.parsed = await team.parser.run(state.requirementText);
      }],
      ['explore', async () => {
        state.observation = await team.explorer.run({
          url: new URL('/login', context.baseUrl).href,
          objective: 'Find the login form',
        });
      }],
      ['plan', async () => {
        state.plan = await team.planner.run({
          parsed: state.parsed!,
          observation: state.observation!,
        });
      }],
      ['generate', async () => {
        state.generated = await team.generator.run({
          scenarios: state.plan!.scenarios,
          observation: state.observation!,
          context,
        });
      }],
      ['validate', async () => {
        const errors = validateGeneratedFile(state.generated!, state.observation!);
        if (errors.length > 0) {
          throw new StopWorkflow(`Generated code rejected: ${errors.join('; ')}`);
        }
      }],
      ['write', async () => {
        const file = state.generated!;
        const summary = `Write ${file.path}\n\n${file.content}`;
        if (!(await this.approve(state, 'write-generated-code', summary))) {
          throw new StopWorkflow('A person rejected writing the generated code');
        }
        await tools.files.writeGenerated(file);
      }],
      ['static-checks', async () => {
        for (const name of ['typecheck', 'lint'] as const) {
          const result = await tools.runner.run(name);
          if (result.exitCode !== 0) {
            await tools.files.removeGenerated(state.generated!.path);
            throw new StopWorkflow(`${name} failed: ${result.output.slice(0, 500)}`);
          }
        }
      }],
      ['execute', async () => {
        const summary = `Run the generated tests against ${context.baseUrl}`;
        if (!(await this.approve(state, 'run-generated-tests', summary))) {
          throw new StopWorkflow('A person rejected running the tests');
        }
        state.firstRun = await tools.runner.runGeneratedTests(state.runId);
      }],
      ['analyse', async () => {
        for (const failure of state.firstRun!.failures) {
          const analysis = await team.analyst.run(failure);
          state.analyses.push({ ...analysis, testTitle: failure.title });
        }
      }],
      ['heal', () => this.heal(state)],
    ];
  }

  private async heal(state: WorkflowState): Promise<void> {
    const failure = state.firstRun!.failures[0];
    const analysis = state.analyses[0];
    if (!failure || !analysis) return;

    const path = state.generated!.path;
    const source = await this.tools.files.readGenerated(path);
    const patch = await this.team.healer.run({ failure, analysis, path, source });
    if (!patch) return;

    state.healing = { ...patch, applied: false };
    const summary = `${patch.explanation}\n- ${patch.oldText}\n+ ${patch.newText}`;
    if (!(await this.approve(state, 'apply-healing-patch', summary))) return;

    await this.tools.runner.preserveArtifacts(state.runId);
    await this.tools.files.writeGenerated({
      path,
      content: source.replace(patch.oldText, () => patch.newText),
    });
    state.healing.applied = true;
    state.finalRun = await this.tools.runner.runGeneratedTests(state.runId);
  }

  async run(state: WorkflowState): Promise<string> {
    try {
      let steps = 0;
      for (const [name, stage] of this.stages(state)) {
        steps += 1;
        if (steps > LIMITS.maxWorkflowSteps) {
          throw new StopWorkflow('Reached the maximum number of workflow steps');
        }
        await stage();
        state.completedStages.push(name);
        await saveState(this.root, state);
      }
      state.status = 'completed';
    } catch (error) {
      state.status = 'stopped';
      state.stopReason = stripVTControlCharacters(
        error instanceof Error ? error.message : String(error),
      );
    } finally {
      this.tools.approver.close();
    }
    await saveState(this.root, state);
    return this.team.reporter.run(state);
  }
}

The try block treats every problem the same way. It marks the run as stopped, saves the reason and still produces a report.

---

🛠️ Add Temporary Agents

Three agents do not exist yet. Create these short placeholders so that the project compiles. You will replace them in Lessons 68, 69 and 70.

Create src/agents/failure-analyst.agent.ts.

import type { FailedTest } from '../core/types.js';
import type { LlmClient } from '../llm/llm-client.js';
import type { FailureAnalysis } from '../schemas/failure.schema.js';

// Temporary version. Lesson 68 replaces this file.
export class FailureAnalystAgent {
  constructor(readonly llm: LlmClient) {}

  async run(failure: FailedTest): Promise<FailureAnalysis> {
    return {
      category: 'unknown',
      summary: `Not analysed yet: ${failure.title}`,
      evidence: [],
      confidence: 'low',
      alternatives: [],
    };
  }
}

Create src/agents/healer.agent.ts.

import type { FailedTest } from '../core/types.js';
import type { LlmClient } from '../llm/llm-client.js';
import type { FailureAnalysis, HealingProposal } from '../schemas/failure.schema.js';

export type HealerInput = {
  failure: FailedTest;
  analysis: FailureAnalysis;
  path: string;
  source: string;
};

// Temporary version. Lesson 69 replaces this file.
export class HealerAgent {
  constructor(readonly llm: LlmClient) {}

  async run(input: HealerInput): Promise<HealingProposal | null> {
    console.error(`Healing is not built yet: ${input.path}`);
    return null;
  }
}

Create src/agents/reporter.agent.ts.

import type { WorkflowState } from '../core/workflow-state.js';

// Temporary version. Lesson 70 replaces this file.
export class ReporterAgent {
  constructor(readonly root: string) {}

  async run(state: WorkflowState): Promise<string> {
    console.log(JSON.stringify(state.firstRun ?? state.stopReason, null, 2));
    return 'printed above';
  }
}

Create src/llm/provider-adapter.ts.

import type { Env } from '../config/env.js';
import { demoFailureResponses, demoResponses } from './demo-responses.js';
import type { LlmClient } from './llm-client.js';
import { MockLlmClient } from './mock-llm.js';

// Temporary version. Lesson 67 adds a real model.
export function createLlmClient(env: Env, demoFailure = false): LlmClient {
  if (env.LLM_PROVIDER !== 'mock') {
    throw new Error('Only the mock model is available so far');
  }
  return new MockLlmClient(demoFailure ? demoFailureResponses : demoResponses);
}

---

📌 Connect Everything

Create src/app.ts.

import { randomUUID } from 'node:crypto';
import { readFile } from 'node:fs/promises';
import { ExplorerAgent } from './agents/explorer.agent.js';
import { FailureAnalystAgent } from './agents/failure-analyst.agent.js';
import { GeneratorAgent } from './agents/generator.agent.js';
import { HealerAgent } from './agents/healer.agent.js';
import { PlannerAgent } from './agents/planner.agent.js';
import { ReporterAgent } from './agents/reporter.agent.js';
import { RequirementParserAgent } from './agents/requirement-parser.agent.js';
import { loadEnv } from './config/env.js';
import { buildProjectContext } from './config/project.js';
import { ConsoleApprover } from './core/approval.js';
import { Orchestrator } from './core/orchestrator.js';
import { createState } from './core/workflow-state.js';
import { createLlmClient } from './llm/provider-adapter.js';
import { PlaywrightBrowserTool } from './tools/browser.tool.js';
import { FileSystemTool } from './tools/filesystem.tool.js';
import { TestRunnerTool } from './tools/test-runner.tool.js';

const root = process.cwd();
const env = loadEnv();
const context = buildProjectContext(env);
const llm = createLlmClient(env, process.argv.includes('--demo-failure'));

const requirementPath = process.argv[2] ?? 'requirements/login.md';
const requirementText = await readFile(requirementPath, 'utf8');
const runId = `run-${randomUUID().slice(0, 8)}`;

const orchestrator = new Orchestrator(
  root,
  context,
  {
    parser: new RequirementParserAgent(llm),
    explorer: new ExplorerAgent(new PlaywrightBrowserTool(env.HEADLESS === 'true'), context.allowedDomains),
    planner: new PlannerAgent(llm),
    generator: new GeneratorAgent(llm),
    analyst: new FailureAnalystAgent(llm),
    healer: new HealerAgent(llm),
    reporter: new ReporterAgent(root),
  },
  {
    files: new FileSystemTool(root),
    runner: new TestRunnerTool(root, env.ALLOW_TEST_EXECUTION === 'true'),
    approver: new ConsoleApprover(),
  },
);

const state = createState(runId, requirementText, llm.providerName);
const reportPath = await orchestrator.run(state);
console.log(`\nRun ${runId}: ${state.status}${state.stopReason ? ` - ${state.stopReason}` : ''}`);
console.log(`Report: ${reportPath}`);

---

🔑 Run the Agent

Start the demo in one terminal.

npm run demo

Run the agent in a second terminal, and type yes at each approval.

npm run agent

The run ends like this, with your own run ID.

Run run-a9ff0e93: completed
Report: printed above

Your project now has a real tests/generated/login.spec.ts, and all three tests passed. If you forget to start the demo, the agent stops safely with a connection error. Use Validate AI-Generated Code as your guide when you read the code at each approval.

---

💡 Test the Workflow

Create tests/unit/orchestrator.test.ts.

import assert from 'node:assert/strict';
import { mkdtemp } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { test } from 'node:test';
import { ExplorerAgent } from '../../src/agents/explorer.agent.js';
import { FailureAnalystAgent } from '../../src/agents/failure-analyst.agent.js';
import { GeneratorAgent } from '../../src/agents/generator.agent.js';
import { HealerAgent } from '../../src/agents/healer.agent.js';
import { PlannerAgent } from '../../src/agents/planner.agent.js';
import { ReporterAgent } from '../../src/agents/reporter.agent.js';
import { RequirementParserAgent } from '../../src/agents/requirement-parser.agent.js';
import { ScriptedApprover, type ApprovalAction } from '../../src/core/approval.js';
import { Orchestrator, type Runner } from '../../src/core/orchestrator.js';
import type { ProjectContext } from '../../src/core/types.js';
import { createState } from '../../src/core/workflow-state.js';
import { demoResponses } from '../../src/llm/demo-responses.js';
import { MockLlmClient } from '../../src/llm/mock-llm.js';
import { FileSystemTool } from '../../src/tools/filesystem.tool.js';

const context = {
  baseUrl: 'http://127.0.0.1:4321',
  allowedDomains: ['127.0.0.1:4321'],
  conventions: [],
} as unknown as ProjectContext;

const browser = {
  inspect: async () => ({
    url: 'http://127.0.0.1:4321/login',
    title: 'Demo',
    ariaSnapshot:
      '- textbox "Email"\n- textbox "Password"\n- button "Sign in"',
  }),
};

async function runWorkflow(answers: Partial<Record<ApprovalAction, boolean>>) {
  const root = await mkdtemp(join(tmpdir(), 'qa-agent-'));
  const llm = new MockLlmClient(demoResponses);
  let testRuns = 0;
  const runner: Runner = {
    run: async () => ({ exitCode: 0, output: '' }),
    preserveArtifacts: async () => {},
    runGeneratedTests: async (runId) => {
      testRuns += 1;
      return {
        runId, startedAt: '', finishedAt: '', exitCode: 0,
        passed: 3, failed: 0, skipped: 0, tracePaths: [], screenshotPaths: [], failures: [],
      };
    },
  };
  const orchestrator = new Orchestrator(
    root,
    context,
    {
      parser: new RequirementParserAgent(llm),
      explorer: new ExplorerAgent(browser, context.allowedDomains),
      planner: new PlannerAgent(llm),
      generator: new GeneratorAgent(llm),
      analyst: new FailureAnalystAgent(llm),
      healer: new HealerAgent(llm),
      reporter: new ReporterAgent(root),
    },
    { files: new FileSystemTool(root), runner, approver: new ScriptedApprover(answers) },
  );
  const state = createState('run-test', 'A user can sign in.', 'mock');
  await orchestrator.run(state);
  return { state, testRuns: () => testRuns };
}

test('completes when a person approves each step', async () => {
  const { state, testRuns } = await runWorkflow({
    'write-generated-code': true,
    'run-generated-tests': true,
  });
  assert.equal(state.status, 'completed');
  assert.equal(testRuns(), 1);
  assert.equal(state.approvals.length, 2);
});

test('stops before writing when approval is refused', async () => {
  const { state, testRuns } = await runWorkflow({ 'write-generated-code': false });
  assert.equal(state.status, 'stopped');
  assert.match(state.stopReason ?? '', /rejected writing/);
  assert.equal(testRuns(), 0);
});

test('never runs tests without the second approval', async () => {
  const { state, testRuns } = await runWorkflow({ 'write-generated-code': true });
  assert.equal(state.status, 'stopped');
  assert.equal(testRuns(), 0);
});

---

🌍 Real-Life Example: The Silent Stop

A workflow fails in its third step and exits quietly. Nobody knows that no tests ran, and the team assumes that all is well. Treating a stop as a recorded outcome, with a reason and a report, makes the failure impossible to miss.

---

🧪 Try It Yourself

1. Create the files and run npm run typecheck.
2. Run the agent and approve both steps.
3. Run it again and type no at the first approval.
4. Confirm that no new file was written.
5. Open artifacts/runs and read a state.json file.

---

📋 Ready-to-Copy Prompt

You are a senior engineer reviewing a workflow orchestrator.

Code:
[paste your orchestrator]

Find any path where a stage could be skipped, an approval bypassed, a
failure hidden or the run left without a final state.

Return a table with: Path, Risk and a Unit Test That Would Prove It.
Do not suggest adding a framework.

---

⚠️ Common Mistakes

• Letting agents call each other directly
• Treating a stopped run as a crash
• Keeping no record of each run
• Testing only the successful path
• Starting the agent before the demo app

---

🎯 Key Takeaways

• The orchestrator owns order, approvals and limits.
• Every stop is a recorded outcome.
• Saved state makes runs explainable.
• A mock LLM lets you test the entire workflow free.

---

🎤 Interview Questions

1. What does an orchestrator do in an AI agent system?

Model answer: It controls the order of stages, passes data between them, enforces approvals and limits, and records the outcome. Individual agents only do their own job.

2. How would you test an orchestrator?

Model answer: I use a mock model, a fake browser and a fake runner. I then assert the path, especially that a refused approval stops the work.

3. Why save state after each stage?

Model answer: It provides an audit trail and shows how far a run got. It also makes resuming possible and helps with debugging.

4. What should happen when a stage fails?

Model answer: The workflow should stop, record the reason and still produce a report. It must not continue with missing data or end silently.

5. Why run the entire workflow with a mock first?

Model answer: It proves the pipeline at no cost and with repeatable results. Then, if a real model causes problems, I know they come from the model.

---

📝 Assignment

Add a requirement-gate stage that stops the workflow when the parser reports more than three ambiguities. Add unit tests for both outcomes, and show the reason in the final output.

---

❓ FAQ

Why not use a workflow framework?

A simple list of stages is enough here, and it is easy to read. Consider a framework when you have a real need that it solves.

Does the agent always need the demo running?

Yes, for this project, because the explorer visits the page. Playwright can start the demo for its own test runs, but the explorer needs it first.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Connect an LLM Safely: Add a Real Model to Your QA Agent.