Agent Workflow State: Save Runs, Traces and Evidence

When something goes wrong in production, your evidence is everything. Good agent workflow state answers what happened, in what order and who approved it. In this lesson you will find a real problem in the project so far, where each run destroys the previous run’s evidence, and fix it.

---

💡 What to Keep for Agent Workflow State

Data	Where it lives	Why
Run status and reason	runs table	Answers “what happened?”
Full workflow state	runs.state_json	Explains each stage
Approvals	approvals table	Shows who decided what
Actions	audit_log table	Provides a permanent record
Reports	reports/ folder	Is shared with readers
Traces and screenshots	artifacts/runs/	Is the evidence for diagnosis

You began saving state in AI Agent Orchestrator. The service now also stores it in the database.

---

🔍 Find the Problem

Look at playwright.config.ts. Results always go to artifacts/test-results and artifacts/results.json. On your laptop that was fine. In a service, tomorrow’s run deletes today’s evidence.

---

🧩 Archive Each Run

Create src/service/artifacts.ts.

import { cp, mkdir, readdir, rm, stat } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { resolve } from 'node:path';

const DAY_MS = 86_400_000;

// The next run overwrites these files, so each run keeps its own copy.
export async function archiveRunArtifacts(root: string, runId: string): Promise<void> {
  const target = resolve(root, 'artifacts/runs', runId, 'final');
  await mkdir(target, { recursive: true });
  for (const name of ['test-results', 'results.json']) {
    const source = resolve(root, 'artifacts', name);
    if (existsSync(source)) {
      await cp(source, resolve(target, name), { recursive: true });
    }
  }
}

export async function deleteExpiredArtifacts(
  root: string,
  maxAgeDays: number,
  now: number = Date.now(),
): Promise<string[]> {
  const folder = resolve(root, 'artifacts/runs');
  if (!existsSync(folder)) {
    return [];
  }
  const deleted: string[] = [];
  for (const name of await readdir(folder)) {
    const path = resolve(folder, name);
    if (now - (await stat(path)).mtimeMs > maxAgeDays * DAY_MS) {
      await rm(path, { recursive: true, force: true });
      deleted.push(name);
    }
  }
  return deleted;
}

Now call it when a run finishes. In src/service/main.ts, add the import, and add this option to the Worker.

onFinished: async (run) => {
  await archiveRunArtifacts(root, run.id);
},

The worker waits for this before it takes the next job, so the next run cannot overwrite the files first.

Create tests/unit/artifacts.test.ts.

import assert from 'node:assert/strict';
import { existsSync } from 'node:fs';
import { mkdir, mkdtemp, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { test } from 'node:test';
import { archiveRunArtifacts, deleteExpiredArtifacts } from '../../src/service/artifacts.js';

test('each run keeps its own copy of the evidence', async () => {
  const root = await mkdtemp(join(tmpdir(), 'qa-agent-'));
  await mkdir(join(root, 'artifacts/test-results'), { recursive: true });
  await writeFile(join(root, 'artifacts/test-results/trace.zip'), 'trace');
  await writeFile(join(root, 'artifacts/results.json'), '{}');
  await archiveRunArtifacts(root, 'run-1');
  assert.ok(existsSync(join(root, 'artifacts/runs/run-1/final/test-results/trace.zip')));
  assert.ok(existsSync(join(root, 'artifacts/runs/run-1/final/results.json')));
});

test('only old runs are deleted', async () => {
  const root = await mkdtemp(join(tmpdir(), 'qa-agent-'));
  await mkdir(join(root, 'artifacts/runs/run-1'), { recursive: true });
  assert.deepEqual(await deleteExpiredArtifacts(root, 30), []);
  const later = Date.now() + 31 * 86_400_000;
  assert.deepEqual(await deleteExpiredArtifacts(root, 30, later), ['run-1']);
  assert.equal(existsSync(join(root, 'artifacts/runs/run-1')), false);
});

---

🛠️ Decide How Long to Keep Things

Data	Example retention	Reason
Traces and screenshots	30 days	Large, and may show page content
Reports	90 days	Useful for comparing runs
Audit log	As long as policy requires	Provides accountability

These are only examples, so follow your organisation’s policy. Call deleteExpiredArtifacts from a scheduled job.

---

📌 Moving to Object Storage

Local folders are enough for one server. When you outgrow them, upload each run’s folder to object storage, such as S3, Google Cloud Storage or Azure Blob Storage. Keep the bucket private, and share files using short-lived links.

---

🔑 Backups

A backup you have never restored is only a hope. For SQLite, use a proper backup method and not a plain file copy of a live database, because recent changes may sit in a separate write-ahead log.

• Schedule backups, and store them away from the server.
• Restore one on a test server, regularly.
• Write down how long a restore takes.

---

🌍 Real-Life Example: The Missing Trace

A release fails on Tuesday, and on Wednesday a developer asks for the Playwright trace. It has gone, because Tuesday night’s run replaced it. With one folder per run, the trace is still there, under that run’s ID.

---

🧪 Try It Yourself

1. Add the archive step and run the unit tests.
2. Complete two runs through the service.
3. Confirm that each has its own final folder.
4. Read a run’s state_json from the database.
5. Decide on retention periods for your team.

---

📋 Ready-to-Copy Prompt

You are a senior engineer reviewing data retention for a QA agent service.

What we store: [list data types and where they live]
Our rules: [paste your retention policy with no confidential detail]

Find any data kept too long or not long enough, any place where sensitive
data could appear and any gap in our backup and restore plan.

Ask me about our legal requirements before suggesting retention periods.

---

⚠️ Common Mistakes

• Letting each run overwrite the last
• Keeping traces forever with no owner
• Never testing a restore
• Making artifact storage public
• Copying a live database file as a backup

---

🎯 Key Takeaways

• Every run needs a complete record.
• Each run must keep its own evidence.
• Set a retention period for each kind of data.
• A backup only counts if you have restored it.

---

🎤 Interview Questions

1. What would you store for each agent run?

Model answer: I would store the status, stop reason, workflow state, approvals, audit events, reports, traces and screenshots, all tied to one run ID.

2. How do you stop artifacts from being overwritten?

Model answer: I archive them to a folder named after the run before the next run starts. The worker waits for that to finish.

3. Why are retention policies important?

Model answer: They control storage costs and reduce the risk from keeping sensitive data. Different data needs different lifetimes.

4. Why might a Playwright trace be sensitive?

Model answer: It can capture page content, network requests and form values. I use synthetic data and restrict who can read traces.

5. How do you know your backups work?

Model answer: I restore them regularly on a separate system and verify the data. An untested backup gives false confidence.

---

📝 Assignment

Add a scheduled call to deleteExpiredArtifacts, and log what it removes. Then write a one-page restore procedure, and test it on a copy of your data.

---

❓ FAQ

Can I store traces in the database?

It is better not to. They are large files, so keep them in storage and save only their location.

When should I use object storage?

Use it when you run more than one server or when local disk space becomes a limit.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Playwright Docker Image: Package Your QA Agent.