AI Agent Observability: Logs, LLM Traces and Alerts

When a service fails at 3 a.m., you cannot attach a debugger to yesterday. AI agent observability means recording enough to understand what happened, while keeping sensitive data out. In this lesson you will add structured logs and metrics, then decide which problems deserve an alert.

---

💡 The Three Parts of AI Agent Observability

Signal	Answers	Example
Logs	What exactly happened?	A model call failed for a run
Metrics	How is it trending?	Failed runs per hour
Traces	Where did the time go?	Which stage of a run was slow

---

🔍 Write Structured Logs

Create src/observability/logger.ts.

import { redact } from '../core/guardrails.js';

export function log(
  level: 'info' | 'warn' | 'error',
  event: string,
  fields: Record<string, unknown> = {},
): void {
  const line = JSON.stringify({ at: new Date().toISOString(), level, event, ...fields });
  console.log(redact(line));
}

Every line is JSON, and every line passes through the redact guardrail. Always include a runId when one exists, so that you can follow a single run from start to finish.

---

🧩 Record Metrics

Create src/observability/metrics.ts.

export class Metrics {
  private readonly counters = new Map<string, number>();

  increment(name: string, labels: Record<string, string> = {}, by = 1): void {
    const text = Object.entries(labels)
      .map(([key, value]) => `${key}="${value}"`)
      .join(',');
    const series = text ? `${name}{${text}}` : name;
    this.counters.set(series, (this.counters.get(series) ?? 0) + by);
  }

  render(): string {
    return [...this.counters].map(([series, value]) => `${series} ${value}`).join('\n') + '\n';
  }
}

The output uses a simple text format that monitoring tools such as Prometheus can read. In our test run, /metrics returned this.

qa_agent_llm_calls_total{outcome="ok"} 3
qa_agent_llm_tokens_total 0
qa_agent_runs_total{status="completed"} 1

The token count is zero because the mock model uses no tokens.

---

🛠️ Record Every Model Call

The wrapper you will build in the next lesson writes one log line for each call.

{
  "level": "info",
  "event": "llm.call",
  "runId": "run-ccd886e6",
  "provider": "mock",
  "prompt": "parse-requirement",
  "promptVersion": "1",
  "durationMs": 2,
  "inputTokens": 0,
  "outputTokens": 0,
  "costUsd": 0,
  "outcome": "ok"
}

As you grow, consider OpenTelemetry, which is a widely used standard for traces and metrics. It has conventions for generative AI calls, so check the current status of those conventions before depending on them. Your own agent’s run state already acts as a simple trace.

---

📌 Choose Your Alerts

An alert should mean that a person needs to act. Too many alerts, and people stop paying attention.

Alert	Why it matters
Service health check fails	Nobody can use it
No successful run in 24 hours	It may have silently stopped
Internal failure rate is high	Something is broken
Daily spending is near its limit	Cost is out of line
A worker is stuck on one job	The queue has stopped moving
Approval waiting too long	A person needs a reminder

---

🔑 Build a Simple Dashboard

Panel	Purpose
Runs by status	Overall health
Model calls and errors	Provider problems
Tokens and cost per day	Spending
Time spent waiting for approval	Team bottlenecks
Evaluation pass rate over time	Quality drift

The AI Test Report explains one run. A dashboard shows the pattern across many.

---

🌍 Real-Life Example: The Quiet Failure

A service reports no errors for a week, and everyone is happy. In fact, a configuration change stopped all runs from starting. There were no errors because nothing happened. An alert for “no successful run in 24 hours” would have found it on day one.

---

🧪 Try It Yourself

1. Create both files.
2. Use log in a short script, with a run ID.
3. Add a password to a log field, and confirm that it is removed.
4. Increment a counter and print render().
5. Write down your three most important alerts.

---

📋 Ready-to-Copy Prompt

You are a site reliability engineer designing monitoring for a QA agent
service.

Available signals:
[list your logs and metrics with no secrets]

Suggest a small set of alerts. For each one, give the condition, why a
person must act and the first thing that person should check.

Do not suggest any alert that nobody would act on.

---

⚠️ Common Mistakes

• Logging free text that cannot be searched
• Writing full prompts and page content to logs
• Alerting on everything
• Only monitoring errors, and missing silence
• Leaving out the run ID

---

🎯 Key Takeaways

• Logs, metrics and traces answer different questions.
• Use structured logs with a run ID.
• Record facts about model calls, not their content.
• Alert only when a person needs to act.

---

🎤 Interview Questions

1. What is the difference between logs, metrics and traces?

Model answer: Logs record individual events, metrics show numbers over time and traces follow one request through its steps.

2. What would you log for each LLM call?

Model answer: I would log the run ID, provider, prompt version, duration, token counts, cost and outcome. I would leave out sensitive content by default.

3. Why is logging prompts risky?

Model answer: Prompts can contain sensitive application data or credentials. Logs are often widely accessible and kept for a long time.

4. How would you detect a service that has silently stopped?

Model answer: I would alert on the absence of successful runs, because a stopped system produces no errors.

5. What makes an alert good?

Model answer: It is actionable, it is rare enough to be taken seriously and it tells the responder where to start.

---

📝 Assignment

Add a metric for approval waiting time. Then design a one-page dashboard and write the first three steps a person should take for each alert.

---

❓ FAQ

Do I need an observability platform?

Not at first. Structured logs and a few metrics go a long way.

Should the metrics endpoint be public?

No. In this project, only an admin token can read it.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Cost Control: Token Budgets and Rate Limits for AI Agents.