AI Agent in Production: What Changes for a QA Agent
Your agent works on your laptop, with you watching every step. Running an AI agent in production is different, because other people depend on it and nobody is watching when it fails at night. Before you deploy anything, you need to know what changes. There is one thing that stays the same: “production” here means the agent is a live service, and it still tests only approved test environments.
📖 What You’ll Learn
- What production means for a QA agent
- What changes when a team uses it
- What still must never change
- How to decide whether you are ready to begin
💡 What an AI Agent in Production Means
Explain Like I’m 5: A Home Kitchen and a Restaurant
Imagine: At home, you cook for yourself and can start again if it burns. A restaurant has many customers, rules and a plan for when the oven breaks.
What it really means: Production means other people rely on your system.
In testing: A team sends requests, and it must keep working without you.
| Question | On your laptop | In production |
|---|---|---|
| Who starts a run? | You, in a terminal | Any approved team member |
| Who approves? | You, by typing yes | A second person, from anywhere |
| Where is the data? | Local files | A database and safe storage |
| What if it crashes? | You restart it | It recovers, and someone is alerted |
| Who pays? | You notice the bill | Budgets stop the spending |
| Who can see secrets? | Only you | Strictly controlled |
🔍 What Must Not Change
Explain Like I’m 5: Same Rules on a Bigger Field
Imagine: When children move to a bigger pitch, the rules of the game stay the same.
What it really means: A bigger system keeps every earlier safety rule.
In testing: Tests still run only against approved environments.
Everything from AI Agent Guardrails still applies. Production adds more ways for things to go wrong, so those rules become more important.
Production does not mean testing production
The word describes how the agent runs. Its targets remain approved test or staging environments, unless a separate policy says otherwise.
🧩 The New Risks
| Risk | Example | Lesson |
|---|---|---|
| Too many users at once | Two runs overwrite each other | 78 |
| A crash during a run | Evidence is lost | 80 |
| Unwanted access | Anyone can start a run | 83 |
| A careless approval | Someone approves their own run | 84 |
| A bad prompt change | Quality drops silently | 85 |
| Nobody notices a failure | No runs for three days | 86 |
| A runaway bill | A loop calls the model all night | 87 |
| No plan for incidents | Nobody knows how to stop it | 88 |
🛠️ Decide Who Owns It
Explain Like I’m 5: Who Feeds the Dog?
Imagine: A family gets a dog. If everyone thinks someone else will feed it, the dog goes hungry.
What it really means: A production system needs named owners.
In testing: One person is on call when the agent fails.
Before you deploy, write down these answers:
- Who owns the service, and who is the backup?
- Who may request a run? Who may approve one?
- What is the agent allowed to do, and on which systems?
- What happens to the data, and how long is it kept?
- How will you measure whether it is worth keeping?
You defined your first scope in the project overview. Now you are extending it to a team.
📌 Set a Simple Service Goal
A service goal states what “working” means. Keep it small at first.
| Measure | Example goal |
|---|---|
| Availability | The API responds 99% of working hours |
| Run success | 95% of runs finish without an internal error |
| Approval wait | Most approvals decided within 4 hours |
| Cost | Under the agreed daily budget |
Choose goals that your team really needs. These are only examples, so measure before you promise anything.
🌍 Real-Life Example: The Laptop That Went on Holiday
A tester’s agent runs nightly from their laptop, and the team comes to depend on the results. When that tester goes on holiday, the laptop is shut down. Nobody notices for a week that no tests have run. The agent was useful, but it was never a service.
🧪 Try It Yourself
- Answer the five ownership questions for your project.
- List three things that could go wrong in your team.
- Write one service goal that you could measure.
- Ask a teammate to challenge your answers.
- Use the prompt below to find what you have missed.
📋 Ready-to-Copy Prompt
You are a senior site reliability engineer reviewing a plan to deploy a QA
agent for a team. It will test only approved test environments.
Current state: [describe how it runs today]
Users: [who will use it]
Systems it can reach: [list them with no secrets]
Tell me what I must decide before deploying, the three biggest risks and
the smallest safe first release. Do not suggest more infrastructure than
this team really needs.⚠️ Common Mistakes
- Thinking production only means putting it on a server
- Having no named owner
- Giving more people access than needed
- Letting a successful demo set expectations
- Promising reliability before measuring it
🎯 Key Takeaways
- Production means that people depend on your agent.
- Every earlier guardrail still applies.
- Each new risk needs a planned response.
- A service needs named owners and measurable goals.
🎤 Interview Questions
1. What does it mean to run an AI agent in production?
Model answer: It means operating it as a reliable service for others, with access control, monitoring, recovery, budgets and owners. It is much more than having working code.
2. Does a production QA agent test the production website?
Model answer: Not by default. It is a live service, but its targets remain approved test or staging environments unless a separate policy permits otherwise.
3. What new risks appear when a team uses an agent?
Model answer: Concurrent runs, crashes during runs, unwanted access, weak approvals, unnoticed quality changes and runaway costs.
4. Why are named owners important?
Model answer: Without them, failures go unnoticed and nobody is responsible for fixing them. An owner and a backup make operations clear.
5. What service goals would you set first?
Model answer: I would measure availability, internal run success, approval waiting time and cost. I would base targets on measurement and on what the team needs.
📝 Assignment
Write a one-page production plan with owners, allowed users, target systems, data retention and two measurable service goals. Review it with someone who would use the service.
❓ FAQ
Does my agent need to be a big system?
No. Level 9 uses one small service first. Grow it only when a real need appears.
Do I have to deploy it at all?
No. A well-built local agent is already valuable. Deploy only if a team will use it.
🚀 Next Steps
Well done on finishing this lesson. Complete the assignment, then continue with AI Agent Architecture for Production: API, Queue, Workers and Storage.