Uncategorized

You have reached the final lesson of the course. A QA mock interview turns everything you have learned into practice under pressure, and it is the safest place to discover your weak spots. In this lesson you will run a full mock interview, learn where the ethical lines are and make a plan for what comes next.

---

💡 Run a Full QA Mock Interview

Round	Time	Focus	Practise with
Introduction	5 minutes	Your story	Your 60-second pitch
Testing	15 minutes	Test design and judgement	Manual testing questions
Automation	15 minutes	Code and tools	SDET questions
AI quality	10 minutes	Evaluation and safety	LLM testing questions
System design	25 minutes	Architecture	AI system design interview
Behavioural	10 minutes	Teamwork and ethics	Your prepared stories

Record yourself, or ask an experienced person to interview you. A human gives you feedback on things a model cannot notice, such as your pace and your confidence.

---

🔍 Your 60-Second Introduction

Use this shape, and fill it with true details.

I am a [role] with [X] years in [domain]. I focus on [strength].
Recently, I [did something specific], which [measured result].
To grow my AI skills, I built [project], with [one safety feature].
I am looking for a role where I can [what you want to do next].

---

🧩 Use AI in Interviews the Right Way

Employers set different rules, so read them and ask if you are unsure.

Usually acceptable	Usually not acceptable, unless allowed
Practising with mock questions	Reading live AI answers during an interview
Researching the company	Hidden tools during a coding test
Improving your own resume wording	Submitting AI work as your own without disclosure
Using AI if the employer invites it	Having anyone or anything impersonate you

---

🛠️ Follow AI Rules at Work

Before using AI at work, make sure that you know the following:

• Which tools are approved?
• What data can be entered into them?
• Who owns the output, and must it be disclosed?
• What review is required?
• Who do you tell if something goes wrong?

Your AI data privacy lesson covers the details. Rules and laws differ by country and industry, so follow your company’s legal and security guidance.

---

📌 Ethics for AI Testers

Principle	In practice
Honesty	Say when AI was used, and never invent results
Accountability	You are responsible for what you approve
Fairness	Look for results that disadvantage people
Privacy	Use synthetic data, and protect people’s information
Safety	Test only what you are authorised to test
Transparency	Report limits and failures openly

Testers often notice problems first. If an AI feature could harm people, raising the concern is part of your job.

---

🔑 Your Final Review

You now know how to use AI in daily testing, test AI products, build an agent, deploy it and present it. Go back through the full course and rate each level from 0 to 3. Revisit any level below 2.

Keep learning in a small and steady way. AI tools change quickly, so each quarter choose one thing to learn, build something small and share what you found.

---

🌍 Real-Life Example: The Hidden Assistant

A candidate uses a hidden assistant in a remote interview, and gets through the first round. In the next one, the interviewer asks them to explain a line from their earlier answer, and they cannot. A candidate who had simply said “I do not know, but here is how I would find out” would have done better.

---

🧪 Try It Yourself

1. Write your 60-second introduction.
2. Run the full mock interview, and record it.
3. Watch it and note three things to improve.
4. Find your company’s AI policy.
5. Score yourself on each level of this course.

---

📋 Ready-to-Copy Prompt

You are a panel of interviewers for this QA role:
[paste the job description with the company name removed]

Run a full mock interview in rounds: introduction, testing, automation,
AI quality, system design and behavioural. Ask one question at a time, and
add follow-ups when my answers are weak.

At the end, give each round a score from 0 to 2 with evidence from my
answers. Give me my three most important things to practise.

---

⚠️ Common Mistakes

• Practising only by reading answers
• Using hidden AI during an interview
• Using AI at work without checking the policy
• Inventing results to sound impressive
• Stopping learning after finishing a course

---

🎯 Key Takeaways

• Rehearse the entire interview under real conditions.
• AI can prepare you, but it must not stand in for you.
• Follow each company’s AI policy.
• You remain accountable for what you approve.

---

🎤 Interview Questions

1. Do you use AI to prepare for interviews?

Model answer: Yes, for practice and feedback. During the interview itself, I follow the company’s rules, and these answers are my own.

2. Your manager asks you to paste customer data into a public AI tool. What do you do?

Model answer: I would explain the risk and refuse, then suggest an approved tool or synthetic data. If needed, I would raise it through the proper channel.

3. You discover that an AI feature treats some users unfairly. What do you do?

Model answer: I would gather evidence, report it clearly and explain the impact. I would recommend that the release be reviewed, and I would not ignore it because it is hard to fix.

4. Who is responsible for a bug that AI missed?

Model answer: The team that used the tool is. AI is one input, so we should improve our process and not blame the tool.

5. Why should we trust your AI-assisted work?

Model answer: Because I verify it, can explain it and I report its limits. I never present output that I have not reviewed.

---

📝 Assignment

Complete a full mock interview with a person, and write down what you learned. Update your resume and portfolio based on it. Then apply for one role that genuinely matches your evidence.

---

❓ FAQ

How many mock interviews should I do?

Keep going until your answers feel natural and your feedback stops repeating. For many people, that means at least three.

What should I do after this course?

Improve your project, keep measuring and share what you learn. Teaching one idea to a colleague is a great next step.

---

🚀 Next Steps

Congratulations on completing the course. Finish the assignment, then use the course syllabus to revisit any level you want to strengthen.

“Design an AI agent that writes and runs tests for our company.” In a senior interview, that may be the entire question. An AI system design interview tests how you think when the problem is open. In this lesson you will follow a clear six-step method, and use the system you built in this course as your worked example.

---

💡 How an AI System Design Interview Works

Step	Time	Goal
1. Clarify	5 minutes	Agree scope and constraints
2. High-level design	10 minutes	Draw the components
3. Deep dive	15 minutes	Explain the hard parts
4. Safety	5 minutes	Show how it stays controlled
5. Evaluation	5 minutes	Prove that it works
6. Trade-offs	5 minutes	Name limits and next steps

---

🔍 Step 1: Clarify

Ask questions, and write down the answers:

• What will it test: web, API or both?
• Who uses it, and how many runs are expected each day?
• What systems may it touch? Is production excluded?
• What data can be sent to a model?
• Who approves generated code?
• What does success look like?

Then state your assumptions, such as: “I will design for web testing on staging, one team first, with human approval.”

---

🧩 Step 2: High-Level Design

Users --> API --> Job queue --> Worker
                                  |
                                  v
                            Orchestrator
             +-----------+--------+---------+-----------+
             |           |        |         |           |
          Parser     Explorer   Planner  Generator   Reporter
                                  |
                  Validator --> Human approval --> Executor
                                                      |
                                            Failure analyst, Healer

Stores: database, artifact storage. Model access through one adapter.

Walk through one request from start to finish. Your AI agent architecture lesson covers each part.

---

🛠️ Step 3: Deep Dive

The interviewer will pick an area. Be ready to go deeper on any of these.

Topic	Key points
Orchestration	Code controls order, and agents stay focused
Contracts	Schemas validate every handoff
Model access	One adapter, with timeouts and validation
Evidence	Locators must come from the real page
Execution	Fixed commands, no shell, restricted environment
Concurrency	One worker first, then isolated workspaces
State	Saved per run, and failed runs are not retried automatically

---

📌 Step 4: Safety

Risk	Control
Unsafe model output	Validation and human review
Prompt injection from pages	Data labelling and narrow tools
Reaching wrong systems	Host allowlist and network limits
Runaway cost	Call, token and daily limits
Hiding real bugs	Healing cannot change assertions
Secret exposure	Secret store, redaction and synthetic data

---

🔑 Step 5: Evaluation

Describe how you would know that it is good. Use a fixed evaluation pack, people to score correctness and safety and a gate that blocks regressions. Track test acceptance, classification accuracy, cost and review time.

---

💡 Step 6: Trade-offs

Decision	Benefit	Cost
Human approval	Safety and trust	Slower throughput
Workflow over free agent	Predictable behaviour	Less flexibility
One database first	Simple to run	Limited scale
Model outside merge checks	Reliable pipeline	Slower quality signal

---

🌍 Real-Life Example: The Candidate Who Asked

One candidate immediately draws twelve agents. Another asks whether production is in scope, learns that it is not and designs a small workflow with approvals. The interviewer then adds a tenfold growth requirement, and the second candidate calmly explains how the queue and workspaces would change.

---

🧪 Try It Yourself

1. Set a 45-minute timer.
2. Design the agent on paper, using the six steps.
3. Explain it aloud to someone.
4. Add a new requirement, such as ten teams.
5. Say what you would change, and why.

---

📋 Ready-to-Copy Prompt

You are a staff engineer running a system design interview.

Question: Design an AI agent that creates and runs tests for our company.

Answer my clarifying questions, but do not design it for me. Add one new
constraint part of the way through.

At the end, score me on requirements, architecture, safety, evaluation and
trade-offs. Point out anything I forgot to ask.

---

⚠️ Common Mistakes

• Designing before asking questions
• Drawing too many agents
• Forgetting about failures and cost
• Never mentioning evaluation
• Treating your first design as the only option

---

🎯 Key Takeaways

• Clarify the problem before you design.
• Walk through one request from end to end.
• Use several independent safety layers.
• Explain your trade-offs, and when you would change them.

---

🎤 Interview Questions

1. How would this design change for ten teams?

Model answer: I would isolate each run’s workspace, move to PostgreSQL and object storage, add several workers and add per-team permissions and budgets.

2. What happens if the model provider goes down?

Model answer: Calls time out and retry within limits. The run stops with a clear reason, and the adapter makes it possible to switch providers after evaluation.

3. How do you stop the agent running forever?

Model answer: I enforce limits on workflow steps, model calls, tokens, command time and approval waiting. Each ends the run safely with a recorded reason.

4. Why not use one powerful agent with all the tools?

Model answer: It would be harder to test and control. Focused stages with narrow permissions make failures easier to find and limit the damage.

5. What would you build first?

Model answer: I would build a reliable test foundation, then typed contracts and a mocked workflow. I would add a real model only when that foundation is proven.

---

📝 Assignment

Write a two-page design for an agent that tests a payment flow in staging. Include assumptions, a diagram, safety controls, evaluation and three trade-offs. Then present it in 20 minutes.

---

❓ FAQ

Do I need to know a specific cloud?

Not usually. Explain the concepts, then mention tools that you have really used.

What if I get stuck?

Say what you are thinking, and ask a clarifying question. Interviewers often help when they can follow your reasoning.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with QA Mock Interview with AI: Ethics, Company Policy and Final Review.

Testing an AI product needs a different way of thinking, and interviewers know it. These LLM testing interview questions cover why old approaches fall short and how you would design something better. Answer in terms of evidence and risk, and use the evaluation work you did in Level 5 as your examples.

---

💡 What LLM Testing Interview Questions Look For

Area	They want to hear
Non-determinism	How you test outputs that vary
Evaluation	How you measure quality
RAG	How you separate retrieval from generation
Safety	How you test harmful behaviour
Release	How you decide that it is good enough

---

🔍 Questions on the Fundamentals

Why can’t you just assert an exact output?

The same input can produce different valid wording. I assert required facts, structure and safety properties, and I score quality across a dataset.

How do you test a non-deterministic system?

I run each case several times, use property checks and semantic scoring and judge pass rates. I also keep the model, prompt and settings recorded for each run.

What is a golden dataset?

It is a reviewed set of inputs and expected qualities. It makes changes comparable over time. I build it from real scenarios and include hard cases, as in Golden Dataset for LLM Evaluation.

How would you test a chatbot?

I would define its purpose and limits, then test accuracy, refusals, tone, safety, data handling and regressions. I would combine automated evaluation with human review.

---

🧩 Questions on Evaluation

What metrics would you use for LLM quality?

It depends on the product. Common ones are faithfulness, relevance, completeness, hallucination rate and refusal accuracy. I add cost and latency, and explain them in LLM Evaluation Metrics for Testers.

Can you use an LLM to judge another LLM?

Yes, as a scalable first pass, with a clear rubric. I check the judge against human labels, because it can be biased or wrong.

How do you detect quality drift?

I run a fixed evaluation set on a schedule and after any change, then compare against a baseline. Providers can change model behaviour.

What would you do if your pass rate was high but users complained?

I would look at real conversations and add the failures to the dataset. My evaluation set clearly did not represent real use.

---

🛠️ Questions on RAG and Agents

How do you test a RAG system?

I test retrieval and generation separately. I check that the right documents are found, then that the answer is grounded in them. See Testing RAG Applications.

An answer is wrong. How do you find out why?

I check whether the right content was retrieved. If it was missing, the problem is retrieval or the data. If it was present, the problem is generation or the prompt.

What is different about testing an AI agent?

Besides the final answer, I test its tool choices, arguments, stop conditions and failure handling. I use fake tools and models to make these repeatable. The approach is in Testing AI Agents.

---

📌 Questions on Safety

What is prompt injection, and how do you test for it?

It is input designed to override a system’s instructions. I test direct attempts and instructions hidden in retrieved content, on systems I am authorised to test. Details are in Prompt Injection Testing.

How do you test for bias?

I compare results for paired inputs that differ only in a sensitive attribute, then review the differences with people who understand the context.

How do you test data leakage?

I use seeded canary values and attempt to extract them. I also check logs and traces for sensitive content.

---

🌍 Real-Life Example: The Perfect Score

A team reports 98% on its evaluation, and users are still unhappy. A tester discovers that the dataset contains only easy questions written by the developers. Adding real and difficult questions drops the score sharply, and the team finally starts fixing the right problems.

---

🧪 Try It Yourself

1. Explain non-determinism in thirty seconds.
2. Design five test cases for a support chatbot.
3. Write a 0 to 2 rubric for one quality.
4. Decide how you would investigate a wrong answer.
5. Run the prompt below.

---

📋 Ready-to-Copy Prompt

You are a senior AI quality engineer interviewing me for an LLM testing role.

Product: [describe an AI feature with no confidential detail]

Ask me one question at a time on evaluation, RAG, agents, safety and release
decisions. Add follow-up questions when my answers are vague.

Then score me against a 0 to 2 rubric, and name the two topics I should
study next.

---

⚠️ Common Mistakes

• Trying to assert exact wording
• Quoting a pass rate without describing the dataset
• Treating LLM judges as always correct
• Forgetting cost and latency
• Testing attacks without permission

---

🎯 Key Takeaways

• Test properties, not exact strings.
• A fixed, realistic dataset makes change visible.
• Separate retrieval from generation.
• Tie every release decision to risk.

---

🎤 Interview Questions

1. How would you decide whether an LLM feature can be released?

Model answer: I would agree thresholds by risk, check evaluation results and safety cases, review failures by hand and plan monitoring and rollback. A single average is never enough.

2. How do you test a model upgrade?

Model answer: I run the same dataset on both versions, compare quality, safety, cost and latency, then review the differences before gradually rolling it out.

3. How do you build a dataset with no production data?

Model answer: I start with requirements, expert-written cases and synthetic edge cases. I add real anonymised failures later, where policy allows it.

4. How would you monitor an LLM in production?

Model answer: I track errors, latency, cost, refusals and user feedback, and sample outputs for review. I also run a scheduled evaluation to detect drift.

5. What is your biggest concern with AI quality today?

Model answer: Fluent answers can hide errors, so teams may trust them too easily. I address that with evidence, human review and honest measurement.

---

📝 Assignment

Write a test strategy for a fictional company knowledge assistant. Include scope, a dataset plan, metrics, safety cases, release criteria and monitoring. Limit it to two pages.

---

❓ FAQ

Do I need to know how to train models?

Usually not for QA roles. Understanding how models behave and how to evaluate them matters more.

Which evaluation tool should I learn?

Learn the concepts first, then one tool such as the one covered in LLM Regression Testing in CI.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with AI System Design Interview: Design a QA Agent Step by Step.

SDET interviews test how you build and defend systems. These SDET AI interview questions go beyond “Have you used Copilot?” into design choices, code review, agents and CI. Each answer shows how a strong candidate reasons. Use the project you built in this course as your evidence, and be ready to change its code on the spot.

---

💡 What SDET AI Interview Questions Test

Area	They are checking
AI-assisted coding	Do you review what is generated?
Framework design	Can you keep a suite maintainable?
Agents and MCP	Do you understand the risks?
CI and reliability	Can you run this at scale?
Live coding	Can you think without relying on an assistant?

---

🔍 Questions on AI-Generated Automation

What do you check in a Playwright test written by AI?

I check locators against the real page, the meaningfulness of assertions, fixed sleeps, hard-coded data and whether the test fails when the feature breaks. I actually break the feature to confirm this.

How do you stop AI from producing brittle tests?

I give it real page evidence and our conventions, then enforce those rules using lint and validation. I prefer role-based locators and web-first assertions.

How do you use AI to investigate a flaky test?

I give it the trace, logs and error, then ask for hypotheses with evidence. I confirm the cause myself and avoid hiding the problem with retries. The method is in Flaky Test Diagnosis with AI.

AI-generated tests all pass. Are you satisfied?

No. Passing shows only that they ran. I would run mutation-style checks by breaking the behaviour, then review traceability to the requirements.

How would you review a large AI-generated pull request?

I would ask for it to be split. A huge diff cannot be reviewed well, so I would check the tests and risky areas first.

---

🧩 Questions on Agents and MCP

What is the difference between a workflow and an agent?

In a workflow, code decides the order of steps. In an agent, the model chooses the next step within limits. I use a workflow when the order is known.

How do you test an LLM-based component without paying for a model?

I place the model behind an interface and use a mock with scripted responses. That lets me test good, malformed and hostile outputs repeatably.

What is MCP, and what are its risks?

MCP is a standard way for AI applications to discover and call tools. The risks include poisoned tool descriptions, excessive permissions and changed tools. I use allowlists and pin versions, as covered in MCP Security for Testers.

How do you make self-healing safe?

Only test defects get a patch, the patch must be small, assertions cannot change and a person approves it. Healing an expected result can hide a real bug. See Self-Healing Tests with AI.

How do you prevent an agent from running dangerous commands?

It can only request tools from a fixed list. Commands use fixed arguments with no shell, a time limit and a limited environment.

---

🛠️ Questions on CI and Reliability

Would you put LLM calls in your merge pipeline?

Not as a blocking check at first. They add cost and variation. I keep mock-based tests in the main pipeline and run model evaluations separately.

How do you measure an AI test generator?

I measure scenario validity, acceptance rate, defects found, review time and cost. A high pass rate alone proves very little.

---

📌 Handling a Live Coding Round

1. Repeat the problem in your own words.
2. Ask about edge cases.
3. Start with the simplest working version.
4. Talk through what you are doing.
5. Test it and explain what you would add.

---

🌍 Real-Life Example: The Mutation Moment

A candidate shows an AI-generated test, then breaks the login button on purpose. The test still passes, and the candidate explains why that is a problem and fixes the assertion. That three-minute demonstration says more than any list of tools.

---

🧪 Try It Yourself

1. Answer five questions aloud, without notes.
2. Break a feature and see whether your test notices.
3. Write a Playwright test without any AI help.
4. Explain one design decision from your agent.
5. Run the mock interview prompt below.

---

📋 Ready-to-Copy Prompt

You are a senior SDET interviewing me for an automation role.

Here is my project's README: [paste with no secrets]

Ask me one technical question at a time about my design decisions, code
quality, agents and CI. Challenge my answers with follow-up questions.

After five questions, tell me which answers sounded memorised and where I
need stronger evidence.

---

⚠️ Common Mistakes

• Treating a passing test as proof of quality
• Being unable to code without an assistant
• Describing agents with no mention of limits
• Giving tool lists instead of decisions
• Not knowing your own project’s weak points

---

🎯 Key Takeaways

• Explain why you made each decision.
• Prove that tests fail when they should.
• Know the risks of agents and MCP.
• Practise coding without AI help.

---

🎤 Interview Questions

1. Design a test strategy for code that is mostly AI-generated.

Model answer: I would raise the bar on review, use contract and behaviour tests, add mutation-style checks and require people to confirm expected results. I would track escaped defects.

2. A model upgrade changed your agent’s behaviour. What do you do?

Model answer: I would roll back if needed, run the evaluation pack against both versions, study the differences and release only when the gate passes.

3. How do you handle secrets in test automation?

Model answer: I use a secret store, synthetic accounts and restricted environments. Generated tests may read only named variables, and logs are redacted.

4. When would you not use AI for automation?

Model answer: For stable deterministic scripts, safety-critical checks without review or any work with forbidden data. Plain code is often the better choice.

5. What was the hardest bug in your agent project?

Model answer: I describe a real one, how a test or review revealed it, how I fixed it and which new check now prevents it.

---

📝 Assignment

Pick three questions from this page and write full answers using your own project. Then do a 45-minute live coding practice with no AI assistance, and review where you got stuck.

---

❓ FAQ

Can I use Copilot in a coding interview?

Only if the interviewer says that you can. Ask at the start, and follow the rules that they give.

What if I built the project with AI help?

That is fine if you are open about it and can explain and change every part of it.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with LLM Testing Interview Questions: RAG, Evaluation and Safety.

Interviewers no longer ask only whether you use AI. They ask how you know that it was right. These manual testing AI interview questions focus on judgement, because that is what makes a manual tester valuable alongside AI. Adapt each answer using your own real examples. For general automation topics, see our AI QA automation interview questions.

---

💡 How to Answer Manual Testing AI Interview Questions

Part	Say
Situation	The project and the risk
Task	What you had to do
Action	What you did, and how you used AI
Result	What changed, with evidence
Learning	What you would do differently

Keep each example to about two minutes. Prepare five stories in advance: a critical bug, a mistake that AI made, a disagreement, a missed defect and an improvement that you led.

---

🔍 Questions on Using AI at Work

How do you use AI in manual testing?

I use it to review stories, draft test ideas and summarise logs. I give it clear context and I review everything against the requirement, because it can miss business rules or invent behaviour.

How do you verify test cases generated by AI?

I check them against acceptance criteria, remove anything invented and look for missing negative, boundary and role-based cases. The checks from How to Review AI-Generated Test Cases are what I use.

Tell me about a time when AI gave you a wrong answer.

I would describe a real case. A good answer names what looked right, how I detected the mistake and what I changed in my process.

What should you never paste into an AI tool?

Customer data, credentials, confidential code and anything that my company’s policy forbids. I use approved tools and synthetic data, as covered in AI Data Privacy for Testers.

Can AI do exploratory testing?

It can suggest charters and ideas, but real exploration depends on noticing the unexpected and following it. I use AI to prepare and to summarise, and I do the investigation myself.

---

🧩 Questions on Testing Skills

How would you use AI to review a user story?

I ask it to find vague words, missing errors and unclear roles. I treat each finding as a question for the team, because it does not know our product decisions.

How do you decide what to test when time is short?

I rank by business impact, likelihood of failure and recent changes. AI can organise the inputs, but I make the final risk decision and agree it with the team.

What makes a bug report good?

Clear steps, the expected and actual results, the environment, evidence and impact. AI can improve the wording, but I confirm that the bug reproduces.

How do you test something with unclear requirements?

I write down my assumptions, ask specific questions and explore the riskiest areas first. I record what I could not confirm.

---

🛠️ Questions on Your Growth

You have not used this AI tool. How would you learn it?

I would be honest, then explain how I learned a similar tool. I would start with a small, safe task and compare the results with my own work.

Will AI replace manual testers?

It will change the work. Routine drafting can be automated, but risk judgement, product understanding and accountability still need people.

What is your weakness with AI?

I give a real one and show what I am doing about it. For example, I may once have trusted fluent output too quickly, so I now check it against a written checklist.

---

🌍 Real-Life Example: The Honest Candidate

Asked about AI, one candidate lists tools for two minutes. Another says, “An assistant gave me test cases for a coupon feature, and it invented a rule that coupons stack. I caught it by checking the story.” The second answer is shorter and far more convincing.

---

🧪 Try It Yourself

1. Write your five prepared stories.
2. Time yourself telling each one.
3. Record your answer to one question.
4. Remove any claim that you could not defend.
5. Practise with a friend.

---

📋 Ready-to-Copy Prompt

You are a QA hiring manager interviewing me for a manual testing role.

Job description: [paste with company name removed]

Ask me one question at a time about testing and how I use AI. After each
answer, tell me what was strong, what was vague and what evidence was
missing. Do not write an answer for me unless I ask.

---

⚠️ Common Mistakes

• Listing tools and giving no example
• Claiming that AI output needs no checking
• Inventing stories
• Forgetting testing fundamentals
• Pretending to know a tool

---

🎯 Key Takeaways

• Structure every example.
• Always explain how you checked AI’s work.
• Use true stories, and prepare them in advance.
• Honesty about gaps earns trust.

---

🎤 Interview Questions

1. A manager wants to replace test design with AI. What do you say?

Model answer: I would agree to use it for drafting and propose a trial with measured quality. I would explain what could be missed and keep human review for important features.

2. AI found no bugs, and you have doubts. What do you do?

Model answer: I would treat it as one input. I would test the riskiest areas myself and look at what information the tool could not have seen.

3. How would you introduce AI to a manual QA team?

Model answer: I would begin with one low-risk task, approved tools and clear rules. We would measure results and share what worked and what failed.

4. How do you measure whether AI helped you?

Model answer: I compare time and quality on similar tasks, including review time and defects missed.

5. What is a hallucination, in testing terms?

Model answer: It is a confident output with no basis in reality, such as a feature that does not exist. I detect it by checking against requirements and the application itself.

---

📝 Assignment

Write out your answers to five questions from this page using your own examples. Practise them aloud three times, and then ask someone to interrupt you with follow-up questions.

---

❓ FAQ

What if I have not used AI at work?

Say so, then describe your practice project and what you learned from it.

Should I memorise model answers?

No. Learn the structure and the reasoning, then use your own experience.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with SDET AI Interview Questions: Automation and Agents with Answers.

Almost every resume now says “AI” somewhere, so the word alone means very little. A strong QA resume with AI skills shows what you did, how you checked it and what changed as a result. In this lesson you will turn vague claims into evidence, and use AI to improve your profile without letting it invent anything.

---

💡 Write Resume Bullets That Prove Something

Use this formula: Action + AI or tool + how you verified it + result.

Weak	Strong
Used AI for testing	Drafted API test cases with an LLM, reviewed each against requirements and reduced design time by 30% over 12 stories
Knows Playwright and AI	Built a Playwright smoke suite, with AI-assisted drafts reviewed against an agreed checklist
Worked on AI agents	Built a QA agent with human approval, allowlists and 32 unit tests, on a demo application

The numbers above are examples only. Use your own measured numbers, or describe the result without one.

---

🔍 Structure a QA Resume with AI Skills

[Name] | [City] | [Email] | [LinkedIn] | [GitHub]

SUMMARY
QA Automation Engineer with [X] years of experience in [domain].
Builds reliable Playwright suites and uses AI with clear review controls.

SKILLS
Testing: test design, API testing, exploratory testing
Automation: Playwright, TypeScript, GitHub Actions
AI in QA: prompt design, LLM evaluation, MCP, human review controls

EXPERIENCE
[Role], [Company], [Dates]
- [Action + tool + how you verified it + result]

PROJECTS
QA Automation AI Agent | [GitHub link]
- [What it does, how it is kept safe and what you measured]

---

🧩 Tailor Each Application

Read the job description and use its real terms where they honestly apply. Automated screening systems and people both look for relevant skills. Do not paste in keywords that you cannot back up, and never hide text in the file. If you are unsure which role to aim for, return to QA Career Path with AI.

---

🛠️ Improve Your LinkedIn Profile

Section	What to do
Headline	Say your role and specialism
About	Tell a short story and link your best project
Featured	Add your repository and demo video
Experience	Reuse your best evidence-based bullets
Posts	Share one specific lesson you learned

A strong headline might be “QA Automation Engineer, Playwright, testing AI features and building safe QA agents”. You can show your QA GitHub portfolio in the Featured section.

---

📌 Use AI on Your Application Safely

AI is useful for tightening language and spotting gaps. It is also happy to invent achievements. Read every line, and remove anything untrue. Delete personal details and employer secrets before you paste anything into a tool.

---

🌍 Real-Life Example: The Invented Metric

A candidate lets an assistant rewrite a resume, and it adds “improved test coverage by 45%”. In the interview, the candidate cannot explain how that was measured, and the rest of the resume loses credibility as well. One invented number can undo everything else.

---

🧪 Try It Yourself

1. Choose one job description.
2. Rewrite three resume bullets using the formula.
3. Mark any number that you cannot prove.
4. Update your LinkedIn headline.
5. Run the prompt below, and check its result.

---

📋 Ready-to-Copy Prompt

You are an experienced technical recruiter for QA roles.

Job description: [paste with company name removed]
My resume: [paste with personal details removed]

Improve clarity and suggest which evidence to move forward.

Rules:
- Do not add any achievement, number, tool or title that is not in my text.
- If a bullet needs a number, write [ADD MEASURED NUMBER] instead.
- List any claim that an interviewer might challenge.

---

⚠️ Common Mistakes

• Listing every AI tool you have ever heard of
• Using results that were never measured
• Sending the same resume to every job
• Letting AI invent achievements
• Pasting confidential work into an AI tool

---

🎯 Key Takeaways

• Show what you did, how you checked it and the result.
• Only list what you can defend.
• Tailor each application honestly.
• AI can improve wording, and you own the facts.

---

🎤 Interview Questions

1. Your resume mentions AI. What did you actually do?

Model answer: I give one specific example, covering the task, the tool, how I checked the output and what changed. I avoid general statements about AI.

2. How did you get this number on your resume?

Model answer: I explain the baseline, how I measured it and the sample size. If it is only an estimate, I say so.

3. Did you use AI to write your resume?

Model answer: I used it to improve the wording, and I checked every fact. Each achievement and number comes from my own work.

4. Which AI tools are you strongest with?

Model answer: I name one or two that I have used on real work, and I explain their strengths and weaknesses. I do not list tools I have only tried once.

5. Why should we hire you over someone with more automation experience?

Model answer: I focus on the role’s needs and on evidence of how I solve problems, including review habits and safety. I am open about what I still need to learn.

---

📝 Assignment

Rewrite your whole resume using the formula, with no unproven numbers. Ask a QA colleague to challenge each bullet, then cut or fix any that they do not believe.

---

❓ FAQ

How long should my resume be?

One page suits early careers and two pages suit experienced testers. Follow local norms.

Should I mention this course?

Yes, as training, but give more space to the project that you built during it.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Manual Testing AI Interview Questions and Model Answers.

A hiring manager might spend only a few minutes on your repository. A strong QA GitHub portfolio makes those minutes count, with a clear story, proof that it works and honest numbers. In this lesson you will prepare your project for public view, starting with the most important check of all: making sure that it contains no secrets.

---

💡 Clean Your QA GitHub Portfolio First

Check these before you make anything public:

• Search your code and Git history for keys, tokens and passwords.
• Confirm that .env was never committed.
• Remove employer code, data, names and screenshots.
• Use only synthetic test data.
• Delete any secret that was ever exposed, and create a new one.

---

🔍 Choose What to Show

Project	What it proves
Playwright framework	Reliable automation
AI evaluation pack	You can test AI products
QA agent with guardrails	You can build safe AI systems

The QA agent project from Level 8 is ideal, because it shows design, testing and safety together.

---

🧩 Write the README

A good README answers a reader’s questions in order. Use this structure.

# QA Automation AI Agent

An AI-assisted QA workflow that turns a requirement into reviewed
Playwright tests, runs them and writes an evidence-based report.

## Demo
[Link to a 3-minute video]

## Why I Built It
[The problem, in two sentences]

## How It Works
[Architecture diagram]

## Safety Design
- Code controls the workflow, not the model
- People approve writing and running generated code
- Test targets are limited by an allowlist

## Quick Start
[Commands that work from a fresh clone]

## Results
[Real numbers, and how you measured them]

## Known Limits
[What it cannot do yet]

Known Limits is the section that many people leave out. It builds more trust than almost anything else.

---

🛠️ Record a Short Demo

1. State the problem in one sentence.
2. Show the requirement.
3. Run the agent, and read one approval.
4. Show one failure and its diagnosis.
5. Reject an approval to show control.
6. Open the report.

The script from AI Agent Acceptance Criteria already gives you this story. Use synthetic data, and hide any personal details on your screen.

---

📌 Show Real Metrics

Weak claim	Strong claim
“Saves huge amounts of time”	“Cut one task from 60 to 25 minutes across 10 timed runs”
“Very accurate”	“Matched human diagnosis in 8 of 10 reviewed failures”
“Fully tested”	“32 unit tests, and one real browser smoke test”

Say how you measured each number, and give the sample size. Small samples are fine if you are open about them. The AI testing ROI lesson shows how to measure fairly.

---

🌍 Real-Life Example: The Honest README

A candidate’s README says that the healer works on only one kind of failure, and that a real provider has been tested on just five requirements. The interviewer is impressed by that honesty, and spends the interview discussing how the candidate would improve it. It becomes the best conversation of the day.

---

🧪 Try It Yourself

1. Search your repository history for secrets.
2. Write your README using the template.
3. Ask a friend to set up the project using only the README.
4. Record a three-minute demo.
5. Pin your best two or three repositories.

---

📋 Ready-to-Copy Prompt

You are a hiring manager reviewing a QA engineer's GitHub project.

README:
[paste your README with no secrets]

Tell me what you understand after 60 seconds, what is unclear and which
claims you do not believe yet. Say what evidence would convince you.

Do not invent results for me. Mark anything that I need to measure.

---

⚠️ Common Mistakes

• Publishing a repository that contains secrets
• Using code that belongs to an employer
• Pinning many unfinished projects
• Making claims without numbers or method
• Hiding what the project cannot do

---

🎯 Key Takeaways

• Check for secrets, including in the history.
• A few complete projects beat many small ones.
• Keep your demo short, and show control.
• Honest metrics and known limits build trust.

---

🎤 Interview Questions

1. Walk me through your portfolio project.

Model answer: I start with the problem, then the architecture, then one design decision and its trade-off. I finish with the results and what I would do next.

2. How did you measure your results?

Model answer: I timed the same tasks with and without the tool and included review time. I state the sample size and know that it is small.

3. How did you make sure that no secrets were exposed?

Model answer: I used environment variables and a gitignore from the start, then scanned the code and its history before publishing. All test data is synthetic.

4. What are the limits of your project?

Model answer: I can name them directly. For example, it covers one workflow and needs human review. I also explain what I would build next.

5. Did you use AI to build it?

Model answer: Yes, and I am open about it. I reviewed the output, wrote tests and can explain each design decision. The tests caught real mistakes along the way.

---

📝 Assignment

Publish one project with a complete README and a demo video. Ask someone who has not seen it to clone it and run it. Fix the first thing that confuses them.

---

❓ FAQ

Can I share a portfolio built with AI help?

Yes, if you say so and can explain it all. Be ready to change the code live.

What if my code is not perfect?

Nobody expects perfection. Clear thinking, tests and honesty matter more.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with QA Resume with AI Skills: Resume and LinkedIn for Testers.

You have learned a lot in this course, but employers cannot see what is inside your head. Your QA career path with AI becomes real when you can show evidence at each stage. In this lesson you will find where you are today, choose your next step and decide exactly what to build to prove it.

---

💡 The QA Career Path with AI

Stage	Main focus	AI skills expected	Evidence to show
Manual QA	Finding important problems	Reviews AI output carefully	Test plans and bug reports
QA Automation	Reliable automated checks	Uses AI to draft and fix tests	A Playwright project
SDET	Test systems and tooling	Builds tools with AI and MCP	A framework or tool
AI QA Engineer	Testing AI products	Evaluation and safety testing	An evaluation pack
QA Lead	People and strategy	Sets safe team practices	Adoption results
QA Architect	Whole-system quality	Designs agents and governance	Architecture decisions

These titles vary between companies, so read each job description closely. The stages are also not a single line. A lead and an architect, for example, are different paths.

---

🔍 What Does Not Change

Curiosity, risk thinking and clear communication stay valuable. You read about this in Will AI Replace Software Testers?. AI changes how the work is done, and your judgement decides whether it was done well.

---

🧩 Map Your Skills

Score each skill from 0 to 3, and write one piece of evidence beside it.

Skill	0	1	2	3
Test design	None	With help	Independently	Teaches others
Automation	None	Edits scripts	Builds suites	Designs frameworks
Using AI safely	None	Basic prompts	Reviews output well	Sets team rules
Testing AI	None	Knows the terms	Runs evaluations	Designs strategy
Building agents	None	Followed a tutorial	Built one and tested it	Runs one in production

---

🛠️ Build a 90-Day Plan

Period	Goal	Output
Days 1 to 30	Close your biggest skill gap	One small finished project
Days 31 to 60	Apply it to real work	One measured improvement
Days 61 to 90	Make it visible	A portfolio item and updated profile

Choose only one target role. If you are moving from manual testing, begin with the AI testing project for beginners. If you already automate, the QA agent project is your strongest portfolio item.

---

📌 Grow Inside Your Current Job

The best evidence often comes from your own work. Offer to reduce one repetitive task, measure the time before and after, and share what you learned. Follow your company’s rules, as covered in AI Data Privacy for Testers.

---

🌍 Real-Life Example: Two Paths

Two manual testers want to move forward. One collects certificates in ten different tools. The other automates one painful smoke test, measures the time saved and shows it to the team. The second tester has a story, a result and a repository, and is the one offered the automation role.

---

🧪 Try It Yourself

1. Find three job descriptions for your target role.
2. Highlight the skills that appear in more than one.
3. Score yourself using the table.
4. Choose your single biggest gap.
5. Write your 90-day plan.

---

📋 Ready-to-Copy Prompt

You are an experienced QA hiring manager and career coach.

Target role: [role]
Job descriptions: [paste with company names removed]
My skills and evidence: [paste your scored table]

Identify my three most important gaps. Suggest one small project for each
that would produce evidence I can show.

Be honest about what I am not ready for. Do not invent salary or market
statistics. Tell me what to check for myself.

---

⚠️ Common Mistakes

• Collecting certificates with no projects
• Learning every new tool that appears
• Abandoning testing fundamentals
• Applying for a title without reading the role
• Making a plan with too many goals

---

🎯 Key Takeaways

• Each stage needs its own evidence.
• Testing judgement matters at every level.
• Score your skills honestly.
• Pick one target and one 90-day plan.

---

🎤 Interview Questions

1. Where do you see your QA career going?

Model answer: I want to grow from automation into building AI-assisted test systems. I have been working towards that by building a small agent with guardrails and measuring its results.

2. How has AI changed the tester’s role?

Model answer: AI speeds up drafting, so more of my time goes to risk analysis, review and evidence. Judgement matters more when more of the work is generated.

3. How do you keep your skills current?

Model answer: I pick one area at a time, build something small and measure it. I check official documentation because AI tools change quickly.

4. What is the difference between a QA automation engineer and an SDET?

Model answer: It varies by company. Generally, an SDET works more on test infrastructure and tools, while an automation engineer focuses on building and maintaining automated tests.

5. Tell me about a skill gap that you closed.

Model answer: I describe the gap, what I built to close it and the measurable result, then what I would do differently. It should be a real example.

---

📝 Assignment

Complete your skills table with evidence, and write your 90-day plan. Share it with a mentor or senior colleague, and change one thing based on their feedback.

---

❓ FAQ

Do I need to learn coding to stay in QA?

Not for every role, but basic automation widens your options. Levels 4 and 8 give you a practical route.

Which AI certificate should I get?

Projects usually tell an employer more. A certificate can help you learn, but it should support your evidence and not replace it.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with QA GitHub Portfolio: README, Demo Video and Real Metrics.

An agent earns its place when its work reaches the tools your team already uses, and when you can prove that it helps. In this final lesson of the level you will file bugs in Jira, open pull requests on GitHub and measure AI testing ROI. An honest calculation includes the costs that are easy to forget.

---

💡 Connect the Agent to Your Tools

Output	Destination	Rule
Confirmed product defect	Jira bug	A person confirms it first
Generated test	GitHub pull request	A person reviews and merges it
Run report	Linked from both	Evidence stays available

---

🔍 File Bugs Without Duplicates

Create src/integrations/jira.ts.

import { createHash } from 'node:crypto';
import { redact } from '../core/guardrails.js';
import type { FailedTest } from '../core/types.js';
import type { FailureAnalysis } from '../schemas/failure.schema.js';
import type { ServiceStore } from '../service/store.js';

export type JiraConfig = { baseUrl: string; email: string; apiToken: string; projectKey: string };

export function buildBugPayload(
  projectKey: string,
  runId: string,
  failure: FailedTest,
  analysis: FailureAnalysis,
) {
  const text = redact(
    [`Run: ${runId}`, `Diagnosis (${analysis.confidence}): ${analysis.summary}`, failure.error].join('\n\n'),
  );
  return {
    fields: {
      project: { key: projectKey },
      issuetype: { name: 'Bug' },
      summary: `[QA Agent] ${failure.title}`.slice(0, 250),
      labels: ['qa-agent'],
      description: {
        type: 'doc',
        version: 1,
        content: [{ type: 'paragraph', content: [{ type: 'text', text }] }],
      },
    },
  };
}

export class JiraClient {
  constructor(
    private readonly config: JiraConfig,
    private readonly store: ServiceStore,
  ) {}

  // Only a confirmed product defect becomes a bug, and each one is filed once.
  async createBugOnce(
    runId: string,
    failure: FailedTest,
    analysis: FailureAnalysis,
    approvedBy: string,
  ): Promise<string> {
    if (analysis.category !== 'product_defect') {
      throw new Error('Only product defects can be filed as bugs');
    }
    const fingerprint = createHash('sha256').update(failure.title + failure.error).digest('hex');
    const settingKey = `jira:${fingerprint}`;
    const existing = this.store.getSetting(settingKey);
    if (existing) {
      return existing;
    }

    const credentials = Buffer.from(`${this.config.email}:${this.config.apiToken}`).toString('base64');
    const response = await fetch(`${this.config.baseUrl}/rest/api/3/issue`, {
      method: 'POST',
      headers: {
        Authorization: `Basic ${credentials}`,
        Accept: 'application/json',
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(buildBugPayload(this.config.projectKey, runId, failure, analysis)),
    });
    if (!response.ok) {
      throw new Error(`Jira returned ${response.status}`);
    }
    const { key } = (await response.json()) as { key: string };
    this.store.setSetting(settingKey, key);
    this.store.addAudit(approvedBy, 'jira.bug_created', runId, key);
    return key;
  }
}

Safeguard	Why
Only product_defect	Test problems are not product bugs
approvedBy is required	A person has confirmed it
Fingerprint	One failure creates one bug
redact	Secrets stay out of tickets

You practised writing bug reports in Write Better Bug Reports with AI. Use a Jira API token for an account that can create issues only in the test project.

---

🧩 Deliver Tests as Pull Requests

Create src/integrations/github.ts.

import { isSafeGeneratedPath } from '../core/guardrails.js';
import type { GeneratedFile } from '../schemas/generation.schema.js';

export type GitHubConfig = {
  apiUrl: string;
  owner: string;
  repo: string;
  token: string;
  baseBranch: string;
};

export class GitHubClient {
  constructor(private readonly config: GitHubConfig) {}

  private async call<T>(method: string, path: string, body?: unknown): Promise<T> {
    const { apiUrl, owner, repo, token } = this.config;
    const response = await fetch(`${apiUrl}/repos/${owner}/${repo}${path}`, {
      method,
      headers: {
        Authorization: `Bearer ${token}`,
        Accept: 'application/vnd.github+json',
        'Content-Type': 'application/json',
      },
      body: body === undefined ? undefined : JSON.stringify(body),
    });
    if (!response.ok) {
      throw new Error(`GitHub ${method} ${path} returned ${response.status}`);
    }
    return (await response.json()) as T;
  }

  // Generated tests reach the main branch only through a reviewed pull request.
  async openPullRequest(runId: string, file: GeneratedFile): Promise<string> {
    if (!isSafeGeneratedPath(file.path)) {
      throw new Error(`Path is not approved: ${file.path}`);
    }
    const { baseBranch } = this.config;
    const branch = `qa-agent/${runId}`;

    const base = await this.call<{ object: { sha: string } }>(
      'GET',
      `/git/ref/heads/${baseBranch}`,
    );
    await this.call('POST', '/git/refs', { ref: `refs/heads/${branch}`, sha: base.object.sha });
    await this.call('PUT', `/contents/${file.path}`, {
      message: `Add generated tests from ${runId}`,
      content: Buffer.from(file.content).toString('base64'),
      branch,
    });
    const pull = await this.call<{ html_url: string }>('POST', '/pulls', {
      title: `QA Agent: generated tests from ${runId}`,
      head: branch,
      base: baseBranch,
      body: 'Generated by the QA agent. Please review the assertions before merging.',
      draft: true,
    });
    return pull.html_url;
  }
}

The pull request is a draft, so it cannot be merged by accident. Use a fine-grained token limited to this one repository, and turn on branch protection for your main branch.

---

🛠️ Test Without Touching Real Systems

Create src/evaluation/roi.ts, which the next section uses. Then create tests/unit/integrations.test.ts. It runs a small local stand-in, so your tests never create a real ticket.

export type RoiInput = {
  runsPerMonth: number;
  minutesWithoutAgent: number;
  reviewMinutesWithAgent: number;
  hourlyCostUsd: number;
  modelCostPerRunUsd: number;
  platformCostPerMonthUsd: number;
  maintenanceHoursPerMonth: number;
};

export function calculateRoi(input: RoiInput) {
  const minutesSaved = input.minutesWithoutAgent - input.reviewMinutesWithAgent;
  const hoursSaved = (input.runsPerMonth * minutesSaved) / 60;
  const benefitUsd = hoursSaved * input.hourlyCostUsd;
  const costUsd =
    input.runsPerMonth * input.modelCostPerRunUsd +
    input.platformCostPerMonthUsd +
    input.maintenanceHoursPerMonth * input.hourlyCostUsd;
  return {
    hoursSaved,
    benefitUsd,
    costUsd,
    netUsd: benefitUsd - costUsd,
    roiPercent: costUsd === 0 ? 0 : Math.round(((benefitUsd - costUsd) / costUsd) * 100),
  };
}

import assert from 'node:assert/strict';
import { createServer, type Server } from 'node:http';
import type { AddressInfo } from 'node:net';
import { after, before, test } from 'node:test';
import { calculateRoi } from '../../src/evaluation/roi.js';
import { GitHubClient } from '../../src/integrations/github.js';
import { buildBugPayload, JiraClient } from '../../src/integrations/jira.js';
import type { FailureAnalysis } from '../../src/schemas/failure.schema.js';
import { ServiceStore } from '../../src/service/store.js';

// A local stand-in, so these tests never touch a real Jira or GitHub.
const calls: string[] = [];
let server: Server;
let url: string;

before(async () => {
  server = createServer((request, response) => {
    calls.push(`${request.method} ${request.url}`);
    response.setHeader('Content-Type', 'application/json');
    response.end(
      JSON.stringify({ key: 'QA-1', object: { sha: 'abc' }, html_url: 'https://example.test/pull/1' }),
    );
  });
  await new Promise<void>((done) => {
    server.listen(0, '127.0.0.1', () => done());
  });
  url = `http://127.0.0.1:${(server.address() as AddressInfo).port}`;
});
after(() => server.close());

const failure = {
  title: 'login fails',
  error: 'password=hunter2 rejected',
  tracePaths: [],
  screenshotPaths: [],
};
const analysis: FailureAnalysis = {
  category: 'product_defect',
  summary: 'The login button is broken.',
  evidence: [],
  confidence: 'high',
  alternatives: [],
};
const jiraConfig = () => ({ baseUrl: url, email: 'a@example.test', apiToken: 't', projectKey: 'QA' });

test('a bug report never contains secrets', () => {
  const payload = JSON.stringify(buildBugPayload('QA', 'run-1', failure, analysis));
  assert.doesNotMatch(payload, /hunter2/);
});

test('the same failure is filed only once', async () => {
  calls.length = 0;
  const jira = new JiraClient(jiraConfig(), new ServiceStore(':memory:'));
  assert.equal(await jira.createBugOnce('run-1', failure, analysis, 'ben'), 'QA-1');
  assert.equal(await jira.createBugOnce('run-2', failure, analysis, 'ben'), 'QA-1');
  assert.equal(calls.length, 1);
});

test('test defects are never filed as product bugs', async () => {
  const jira = new JiraClient(jiraConfig(), new ServiceStore(':memory:'));
  await assert.rejects(
    jira.createBugOnce('run-1', failure, { ...analysis, category: 'test_defect' }, 'ben'),
    /Only product defects/,
  );
});

test('generated tests arrive as a pull request, in the right order', async () => {
  calls.length = 0;
  const github = new GitHubClient({ apiUrl: url, owner: 'o', repo: 'r', token: 't', baseBranch: 'main' });
  await github.openPullRequest('run-1', { path: 'tests/generated/login.spec.ts', content: 'x' });
  assert.deepEqual(calls, [
    'GET /repos/o/r/git/ref/heads/main',
    'POST /repos/o/r/git/refs',
    'PUT /repos/o/r/contents/tests/generated/login.spec.ts',
    'POST /repos/o/r/pulls',
  ]);
});

test('calculates return on investment', () => {
  const result = calculateRoi({
    runsPerMonth: 40,
    minutesWithoutAgent: 60,
    reviewMinutesWithAgent: 15,
    hourlyCostUsd: 30,
    modelCostPerRunUsd: 0.5,
    platformCostPerMonthUsd: 50,
    maintenanceHoursPerMonth: 4,
  });
  assert.equal(result.hoursSaved, 30);
  assert.equal(result.benefitUsd, 900);
  assert.equal(result.costUsd, 190);
  assert.equal(result.netUsd, 710);
});

After these pass, try the integrations against a sandbox Jira project and a practice repository, and read each provider’s current API documentation first.

---

📌 Measure AI Testing ROI Honestly

ROI compares what you gain with what you spend. This calculation uses example figures.

Input	Example
Runs per month	40
Minutes per task without the agent	60
Review minutes with the agent	15
Hourly cost	$30
Model cost per run	$0.50
Platform cost per month	$50
Maintenance hours per month	4

Result	Value
Hours saved	30
Benefit	$900
Cost	$190
Net	$710

These numbers are only an illustration. Use your own measurements, and find them by timing the same tasks with and without the agent.

---

🔑 Measure More Than Money

Measure	What it tells you
Test acceptance rate	Whether the output is usable
Defects found by agent tests	Whether the tests catch real problems
False bugs filed	Whether people’s time is being wasted
Coverage of agreed requirements	Whether it is closing gaps
Tester feedback	Whether people want to keep using it

Time saved is only valuable if quality stays the same or improves. Your metrics from the AI agent CI pipeline feed directly into this.

---

🌍 Real-Life Example: The Number That Fell Apart

A team claims that its agent saves 200 hours a month. Someone asks about review time, and it turns out the generated tests needed almost as long to fix as they would have taken to write. A proper measurement shows a smaller but real saving, and it is one the team can defend.

---

🧪 Try It Yourself

1. Create the files, and run the unit tests.
2. Time one task with and without your agent.
3. Fill in the ROI calculation with your numbers.
4. List every cost that you might have forgotten.
5. Write one honest sentence about the value.

---

📋 Ready-to-Copy Prompt

You are a sceptical finance partner reviewing the ROI of a QA AI agent.

Measurements:
[paste your real numbers and how you measured them]

Find any cost I left out, any benefit I overstated and any figure that
was estimated instead of measured. Do not change or invent numbers.

Then write a short, honest summary for a director.

---

⚠️ Common Mistakes

• Filing every failure as a product bug
• Giving integration tokens more power than needed
• Letting the agent merge its own code
• Counting time saved but not review time
• Reporting estimates as if they were measurements

---

🎯 Key Takeaways

• Deliver results into the tools people already use.
• Prevent duplicate tickets with a fingerprint.
• Generated code arrives as a reviewable pull request.
• Honest ROI includes every real cost.

---

🎤 Interview Questions

1. How would you integrate an AI agent with Jira?

Model answer: I would file only human-confirmed product defects, with redacted evidence and a fingerprint to prevent duplicates, using a narrowly scoped token.

2. How should an agent deliver generated tests?

Model answer: It should open a draft pull request. People review and merge it, and branch protection means the agent cannot merge its own work.

3. How do you measure ROI for an AI testing tool?

Model answer: I compare the measured time for the same tasks, then subtract model, platform, review and maintenance costs. I also track quality.

4. What costs are often forgotten?

Model answer: Human review, fixing poor output, maintenance, incidents and training. They can remove much of the apparent saving.

5. How do you avoid duplicate bugs?

Model answer: I create a fingerprint from stable failure details, record it when a bug is filed and return the existing ticket if the same failure comes back.

---

📝 Assignment

Connect your agent to a sandbox Jira project and a practice repository. Measure five real tasks with and without it. Write a one-page summary of the value, including at least one thing that did not work.

---

❓ FAQ

Should the agent close Jira tickets too?

Not at first. Creating a ticket is easy to undo. Closing work needs a much higher level of trust.

What if the ROI is negative?

That is a useful result. Narrow the agent to the tasks where it helps, or stop using it.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with QA Career Path with AI: From Manual Tester to QA Architect.

Launching to everyone on day one is how small problems become big ones. AI agent production readiness means proving that the system is safe to operate, then releasing it to a few people at a time. In this lesson you will add an emergency pause switch, finish the service and write the runbook you hope never to need.

---

💡 The AI Agent Production Readiness Review

Area	Evidence you need
Safety	Allowlists tested, and approvals required
Security	Token tests pass, and secrets are not in the image
Reliability	Crash and restart behaviour tested
Data	A backup has been restored
Cost	Budgets tested, and a provider cap is set
Monitoring	Alerts fire, and someone receives them
Quality	Evaluation gate passes
Operations	Owner named, and the runbook has been rehearsed

You started this kind of checking with AI Agent Acceptance Criteria. This review extends it to a live service.

---

🔍 Add the Emergency Pause

Create src/service/ops-routes.ts.

import express, { type Express } from 'express';
import * as z from 'zod';
import { log } from '../observability/logger.js';
import type { Metrics } from '../observability/metrics.js';
import type { Guard } from './api.js';
import type { Actor } from './security.js';
import type { ServiceStore } from './store.js';

const PauseSchema = z.object({ paused: z.boolean(), reason: z.string().min(1).max(500) });

export function mountOpsRoutes(
  app: Express,
  guard: Guard,
  store: ServiceStore,
  metrics: Metrics,
): void {
  app.get('/metrics', guard('ops:manage'), (_request, response) => {
    response.type('text/plain').send(metrics.render());
  });

  app.post('/v1/ops/pause', guard('ops:manage'), express.json(), (request, response) => {
    const body = PauseSchema.safeParse(request.body);
    if (!body.success) {
      response.status(400).json({ error: 'paused and reason are required' });
      return;
    }
    const actor = response.locals.actor as Actor;
    store.setSetting('paused', String(body.data.paused));
    store.addAudit(actor.id, body.data.paused ? 'ops.paused' : 'ops.resumed', undefined, body.data.reason);
    log('warn', 'ops.pause_changed', { actor: actor.id, paused: body.data.paused });
    response.json({ paused: body.data.paused });
  });
}

The Worker already checks isPaused before it takes a job. A paused service still answers status requests, so people can see what is going on. To pause it:

curl -X POST http://127.0.0.1:8080/v1/ops/pause -H "Authorization: Bearer YOUR-ADMIN-TOKEN" -H "Content-Type: application/json" -d "{\"paused\":true,\"reason\":\"Investigating bad results\"}"

---

🧩 Finish the Service

Replace src/service/main.ts with the final version, which brings together everything from this level, including AI Agent Access Control.

import { randomUUID } from 'node:crypto';
import { loadEnv } from '../config/env.js';
import { createSlackNotifier, mountSlackRoutes } from '../integrations/slack.js';
import { GuardedLlmClient } from '../llm/guarded-llm.js';
import { usageMeter } from '../llm/usage.js';
import { log } from '../observability/logger.js';
import { Metrics } from '../observability/metrics.js';
import { createApi } from './api.js';
import { archiveRunArtifacts } from './artifacts.js';
import { loadServiceConfig } from './config.js';
import { FixedWindowLimiter, StoredSpendLedger } from './limits.js';
import { mountOpsRoutes } from './ops-routes.js';
import { createWorkflowRunner } from './run-workflow.js';
import { LocalDevSecurity } from './security.js';
import { ServiceStore } from './store.js';
import { TokenSecurity } from './token-security.js';
import { Worker } from './worker.js';

const root = process.cwd();
const env = loadEnv();
const config = loadServiceConfig();
const store = new ServiceStore(config.DATABASE_PATH);
const metrics = new Metrics();
const ledger = new StoredSpendLedger(store);
const limiter = new FixedWindowLimiter(config.RUNS_PER_HOUR_PER_USER, 3_600_000);

if (config.AUTH_MODE === 'local-dev' && config.SERVICE_HOST !== '127.0.0.1') {
  throw new Error('AUTH_MODE=local-dev is allowed only on 127.0.0.1');
}

const runWorkflow = createWorkflowRunner(root, env, (client, run) => {
  usageMeter.reset();
  return new GuardedLlmClient(
    client,
    usageMeter,
    {
      maxCallsPerRun: config.MAX_LLM_CALLS_PER_RUN,
      maxTokensPerRun: config.MAX_TOKENS_PER_RUN,
      maxDailyCostUsd: config.MAX_DAILY_COST_USD,
      inputCostPerMillionUsd: config.INPUT_COST_PER_MILLION_USD,
      outputCostPerMillionUsd: config.OUTPUT_COST_PER_MILLION_USD,
    },
    ledger,
    metrics,
    run.id,
  );
});

const worker = new Worker(store, runWorkflow, {
  approvalTimeoutMs: config.APPROVAL_TIMEOUT_MINUTES * 60_000,
  notify: config.SLACK_WEBHOOK_URL ? createSlackNotifier(config.SLACK_WEBHOOK_URL) : undefined,
  onFinished: async (run, status) => {
    await archiveRunArtifacts(root, run.id);
    metrics.increment('qa_agent_runs_total', { status });
    log('info', 'run.finished', { runId: run.id, status });
  },
});

const app = createApi({
  root,
  store,
  security: config.AUTH_MODE === 'token' ? new TokenSecurity(store) : new LocalDevSecurity(),
  environments: ['demo'],
  allowRun: (actorId) => limiter.allow(actorId),
  mount: (server, guard) => {
    mountOpsRoutes(server, guard, store, metrics);
    if (config.SLACK_SIGNING_SECRET) {
      const approvers = JSON.parse(config.SLACK_APPROVERS) as Record<string, string>;
      mountSlackRoutes(server, store, config.SLACK_SIGNING_SECRET, approvers);
    }
  },
});

const server = app.listen(config.SERVICE_PORT, config.SERVICE_HOST, () => {
  log('info', 'service.started', { instance: randomUUID(), port: config.SERVICE_PORT });
});

void worker.start();

for (const signal of ['SIGTERM', 'SIGINT'] as const) {
  process.on(signal, () => {
    log('warn', 'service.stopping', { signal });
    worker.stop();
    server.close(() => process.exit(0));
  });
}

Notice the check near the top. The service refuses to start with local security on anything but your own machine. The final section shuts down gracefully when the platform asks it to stop.

---

🛠️ Roll Out in Stages

Stage	Who	Move on when
Shadow	The owner only, and results are not used	Runs complete reliably
Pilot	One team, one workflow	People accept its output
Limited	A few teams	Cost and support are manageable
General	Everyone approved	Service goals have been met

Write down what will make you stop as well as what will make you continue.

---

📌 Write the Incident Runbook

A runbook tells a tired person exactly what to do.

Step	Action
1. Stop the harm	Pause the service
2. Tell people	Post in the agreed channel
3. Save the evidence	Do not delete artifacts or logs
4. Find the scope	Which runs, systems and data were affected?
5. Fix or roll back	Redeploy the last good release
6. Rotate secrets	If any exposure is possible
7. Resume carefully	Unpause and watch the first runs
8. Review	Write a blameless review and add a test

Symptom	First check
Runs stuck in queued	Is the service paused? Is the worker alive?
Runs interrupted	Did the service restart or run out of memory?
Many model errors	Provider status, key and budget
Browser crashes	Memory and shared memory limits
Unexpected site visited	Pause, and treat it as a security incident

---

🌍 Real-Life Example: The Friday Launch

A team opens an agent to the whole company on a Friday afternoon. A prompt problem floods Jira with duplicate bugs, and the only person who knows how to stop it has gone home. A pilot group, a pause switch and a runbook would have made that a small and boring event.

---

🧪 Try It Yourself

1. Create the files, and start the final service.
2. Pause it, queue a run and confirm that it waits.
3. Resume it and watch the run start.
4. Fill in the readiness table with real evidence.
5. Rehearse the runbook with a colleague.

---

📋 Ready-to-Copy Prompt

You are running a production readiness review for a QA agent service.

Evidence for each area:
[paste your completed review with no secrets]

For each area, say whether the evidence is enough. List what could still
cause an incident on launch day, and ask me the five hardest questions.

Do not accept anything marked as done unless the evidence supports it.

---

⚠️ Common Mistakes

• Launching to everyone at once
• Having no way to stop the agent quickly
• Writing a runbook that nobody has rehearsed
• Deleting evidence during an incident
• Launching just before a weekend

---

🎯 Key Takeaways

• Readiness is proven with evidence.
• Roll out to small groups first.
• A pause switch must work without a deployment.
• Rehearse your incident response.

---

🎤 Interview Questions

1. How would you roll out an AI agent?

Model answer: I would go from shadow mode to a pilot, then limited and general release. Each stage has written criteria for moving on or stopping.

2. What is a kill switch?

Model answer: It is a fast way to stop the system without deploying. Here, an admin pauses the worker, so no new runs start.

3. What is the first step in an incident?

Model answer: Stop further harm, then communicate and preserve evidence. Understanding the cause comes after containment.

4. What should a production readiness review cover?

Model answer: It should cover safety, security, reliability, data, cost, monitoring, quality and operations, each with evidence.

5. What is a blameless review?

Model answer: It focuses on how the system allowed a failure, and not on who to blame. People speak openly, and the system improves.

---

📝 Assignment

Complete the readiness review for your service, and write a one-page runbook. Run a practice incident in which the agent visits an unapproved site, and record what you would improve.

---

❓ FAQ

Does a pause stop a run already in progress?

No. It stops new runs. To stop one immediately, stop the service, and the run will be marked as interrupted.

How long should a pilot last?

Long enough to see real usage and at least one failure. For many teams, that is a few weeks.

---

🚀 Next Steps

Well done on finishing this lesson. Complete the assignment, then continue with Measure AI Testing ROI: Jira and GitHub Integration for QA Agents.