LLM Structured Output: Validate AI Responses with a Schema

Your code cannot safely act on a friendly paragraph from an AI model. It needs data in a known shape. LLM structured output means asking the model for that shape, and then checking that it really arrived. In this lesson you will write a validator and learn what it can and cannot prove.

---

💡 What Is LLM Structured Output?

Free text	Structured output
“Maybe check login, it is quite important…”	Fields with known names and types
Needs a person to read it	Code can process it
Hard to validate	Each field can be checked

You first met this idea in Gherkin Test Cases with AI. Now your code will enforce it.

---

🔍 Define the Shape

type TestScenario = {
  id: string;
  title: string;
  priority: 'high' | 'medium' | 'low';
  steps: string[];
  expectedResult: string;
};

This type helps you while coding, but it disappears when the program runs. AI responses arrive at runtime, so you also need a runtime check.

---

🧩 Write a Validator

This example has no dependencies, so that you can see each rule.

const PRIORITIES = ['high', 'medium', 'low'];

function validateScenario(data: unknown): string[] {
  if (typeof data !== 'object' || data === null) {
    return ['Response must be an object'];
  }

  const scenario = data as Record<string, unknown>;
  const errors: string[] = [];

  if (typeof scenario.id !== 'string' || scenario.id === '') {
    errors.push('id must be a non-empty string');
  }
  if (typeof scenario.title !== 'string' || scenario.title === '') {
    errors.push('title must be a non-empty string');
  }
  if (!PRIORITIES.includes(scenario.priority as string)) {
    errors.push('priority must be high, medium or low');
  }
  if (!Array.isArray(scenario.steps) || scenario.steps.length === 0) {
    errors.push('steps must be a non-empty list');
  }
  if (typeof scenario.expectedResult !== 'string') {
    errors.push('expectedResult must be a string');
  }

  return errors;
}

function checkResponse(text: string): string[] {
  try {
    return validateScenario(JSON.parse(text));
  } catch {
    return ['Response is not valid JSON'];
  }
}

const good = JSON.stringify({
  id: 'TC-01',
  title: 'Login with a valid user',
  priority: 'high',
  steps: ['Open login page', 'Enter valid details', 'Click Login'],
  expectedResult: 'User sees the dashboard',
});

const bad = JSON.stringify({
  id: 'TC-02',
  title: 'Logout',
  priority: 'urgent',
  steps: [],
  expectedResult: 'User sees the login page',
});

console.log(`Good: ${checkResponse(good).length} errors`);
console.log(`Bad: ${checkResponse(bad).join('; ')}`);
console.log(`Broken: ${checkResponse('Sure! Here is your JSON').join('; ')}`);

Good: 0 errors
Bad: priority must be high, medium or low; steps must be a non-empty list
Broken: Response is not valid JSON

In real projects, a schema library such as Zod or Pydantic saves you from writing these checks by hand. Many AI providers also offer a structured output mode. Keep your own validation anyway, because your code is the last check.

---

🛠️ A Valid Response Can Still Be Wrong

Check type	Question	QA example
Schema	Is the shape right?	steps is a non-empty list
Semantic rule	Is the content allowed?	Each ID is unique, and each URL is approved
Evidence	Does it match reality?	The selector exists on the real page
Human review	Is it the right test?	A tester approves the scenarios

---

📌 What Should Happen When Validation Fails?

Reject the response, and never pass it on to the next step. You can send the errors back to the model and ask for a corrected version. Limit those retries, because some failures will never fix themselves. You will build this in the next lesson.

---

🌍 Real-Life Example: The Silent Empty List

A model returns a scenario with an empty list of steps. The JSON is valid, so a careless pipeline accepts it. The generated test has no actions and passes every time. A rule that rejects empty steps stops this false confidence early.

---

🧪 Try It Yourself

1. Run the validator from the example.
2. Remove expectedResult from the good scenario and check the error.
3. Add a rule that an ID must start with TC-.
4. Paste a real AI response from a few-shot test case prompt.
5. Find a response that passes the schema but is still a poor test.

---

📋 Ready-to-Copy Prompt

You are a QA test designer.

Requirement:
[paste a requirement with no confidential data]

Return only a JSON array of test scenarios. Do not use markdown fences.

Each scenario must contain:
- id: string starting with TC-
- title: string
- priority: high, medium or low
- steps: non-empty array of strings
- expectedResult: string

If information is missing, return an empty array and do not invent details.

---

⚠️ Common Mistakes

• Trusting a TypeScript type to check data at runtime
• Calling JSON.parse without handling errors
• Passing invalid output to the next step
• Retrying forever when validation fails
• Treating a schema pass as proof of quality

---

🎯 Key Takeaways

• Structured output gives your code a known shape.
• Validate every AI response at runtime.
• Schemas check shape, not truth.
• Add semantic rules, evidence and human review.

---

🎤 Interview Questions

1. What is structured output from an LLM?

Model answer: It is a response in a defined format, usually JSON, with expected fields and types. It lets code process the result without trying to interpret free text.

2. Why is a TypeScript type not enough to validate model output?

Model answer: Types are removed at compile time, while model responses arrive at runtime. I need a runtime check, using either a validator or a schema library such as Zod.

3. What are the limits of schema validation?

Model answer: A schema confirms structure and allowed values, but not correctness. The output can be well formed while covering the wrong behaviour or using a selector that does not exist.

4. What should happen when the model returns invalid JSON?

Model answer: The system should reject it and record the failure. It can retry with the validation errors a limited number of times, and then stop rather than continuing with bad data.

5. What are semantic checks?

Model answer: They check meaning or business rules beyond the shape, such as unique IDs, approved URLs and steps that match the requirement. They catch problems that a basic schema misses.

---

📝 Assignment

Write a validator for a failure analysis response with category, confidence, evidence and suggestedFix. Allow only known categories, and require at least one evidence item. Write tests for four bad responses.

---

❓ FAQ

Should I write validators by hand?

Learn with a small one, then use a schema library in real projects. It gives clearer errors and less code to maintain.

Does a provider’s JSON mode remove the need for validation?

No. It can make the format more reliable, but your code should still check the response and apply your own rules.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Provider Adapter: Switch AI Models Without Rewriting Code.