How LLMs Work: Tokens, Context and Why Answers Change
Ask an AI assistant the same question twice and you may get two different answers. To a tester, that looks like a flaky test. In this lesson you will learn how LLMs work and why this happens, using four simple ideas: tokens, context, temperature and training cutoff.
📖 What You’ll Learn
- How an LLM produces an answer, one small piece at a time
- What tokens and the context window are, and why they matter
- Why the same prompt can give different output
- What this means for how you use AI in testing
💡 How LLMs Work: One Small Piece at a Time
Explain Like I’m 5: How an LLM Answers
Imagine: You and a friend play a game. Your friend says, “The cat sat on the…” and you shout, “mat!” You keep going, one word at a time, until the sentence ends.
What it really means: The model keeps guessing the next small piece of text.
In testing: It may guess an error message your app never shows.
An LLM does not look up an answer in a database. It predicts the next small piece of text, adds it to the answer, and predicts again until it finishes.
Complete this sentence: “When the password is wrong, the login page should show an error ___.” You probably thought of “message”. An LLM does something similar, using patterns learned from huge amounts of text.
The key insight
An LLM predicts what text is likely to come next. Likely is not the same as true. Keep this in mind whenever you review AI output.
🔍 Tokens: How the Model Reads Text
Explain Like I’m 5: Token
Imagine: A big LEGO house is built from small bricks. A sentence is built from small pieces too. Those pieces are tokens.
What it really means: Tokens are the small chunks of text the model reads and writes.
In testing: A long log has many tokens, so it costs more and fills up the model’s space faster.
A token is a small chunk of text, often a whole short word or part of a longer one. In English, one token averages roughly three quarters of a word, but the exact split depends on the model. Tokens matter for two reasons:
- Paid AI services usually charge by the number of tokens.
- Every model has a limit on how many tokens it can handle at once.
🧩 Context Window: The Model’s Working Memory
Explain Like I’m 5: Context Window
Imagine: You have a small desk. Only so many papers fit on it. When you add a new paper, an old one may fall off.
What it really means: It is the amount of text the model can look at in one go.
In testing: In a very long chat, your first requirement may fall off the desk.
The context window is the amount of text a model can consider at one time. It holds your instructions, your conversation so far, any files you shared and the answer being written.
| Situation | What can happen | What a tester should do |
|---|---|---|
| A very long chat | Early details may be dropped or ignored | Start a new chat for each task |
| A huge log file pasted in | The important line may get lost | Share only the relevant section |
| A new chat is opened | The model normally knows nothing from the old chat unless a memory feature is on | Paste the key context again |
🛠️ Temperature: Why Answers Change
Explain Like I’m 5: Temperature
Imagine: At the ice-cream shop, one child always picks vanilla. Another child picks a new flavour every time. Low temperature is the vanilla child. High temperature is the adventurous child.
What it really means: It is a setting that controls how much variety the answers have.
In testing: Use low for test data and higher for unusual test ideas.
At each step, the model has many possible next words. Some are more likely than others. It does not always choose the most likely one. A setting called temperature controls how much variety is allowed.
| Temperature | Behaviour | Good for |
|---|---|---|
| Low | More focused and repeatable | Structured test data, code |
| High | More varied and creative | Brainstorming unusual test ideas |
Most chat apps do not show this setting, but developers can set it when using an API. Even at a low setting, answers may not be perfectly identical, so never design a check that depends on exact AI wording.
📌 Training Cutoff: The Model’s Knowledge Has a Date
Explain Like I’m 5: Training Cutoff
Imagine: A storybook printed last year cannot tell you what happened yesterday.
What it really means: The model only learned from information up to a certain date.
In testing: It may not know the newest Playwright version.
A model learns from data collected up to a certain date, called its training cutoff. It may not know about the latest Playwright release, and it may suggest a method that has since changed. Some assistants can search the web, but you should still check the official documentation or a current guide such as our Playwright with TypeScript course.
🔑 What the Model Does Not Know
An LLM has never seen your application unless you give it the details. It does not know your business rules, your test environment or last week’s defect. A vague prompt forces it to guess, and a guess can look very convincing. The next lesson shows how these AI hallucinations in software testing appear.
🌍 Real-Life Example: Two Prompts, Two Results
Prompt A: “Write test cases for login.”
The model must guess the fields, the rules and the format, so you get generic cases.
Prompt B: “Write test cases for a login form with email and password. Passwords need 8 to 20 characters. The account locks after 5 failed attempts. Return a table with ID, steps, test data and expected result.”
The model now has real context, so boundary cases such as 7, 8, 20 and 21 characters become much more likely. You will master this skill in Level 2.
🧪 Try It Yourself
In this experiment, you will see AI answers change with your own eyes.
- Open three new chats in an AI assistant.
- Paste the prompt below into each one, without changing it.
- Record the scenarios from each chat in a table.
- Mark which scenarios appear in all three chats, and which appear in only one.
- Combine the best of them into one list and add anything all three missed.
📋 Ready-to-Copy Prompt
You are a senior QA engineer.
Feature: A coupon code field on a checkout page.
Rules:
- One coupon per order
- Coupons expire on a set date
- Minimum order value is 500
List the 10 most important test scenarios, one line each.
Label each one as Positive, Negative or Boundary.
Do not add rules that I have not given you. List any questions
you would ask the product owner at the end.⚠️ Common Mistakes
- Continuing one long chat across unrelated tasks
- Pasting a huge file and expecting the model to find the single important line
- Assuming the model knows your application or the latest tool version
- Treating one AI answer as complete coverage
- Writing automated checks that compare exact AI wording
🎯 Key Takeaways
- LLMs predict text token by token, so likely output is not guaranteed to be true.
- The context window limits what the model can consider at one time.
- Output can vary between runs, so review every result and never rely on exact wording.
- Better context produces better answers.
🎤 Interview Questions
1. What is a token in an LLM?
Model answer: A token is a small unit of text that the model processes, often a short word or a part of a longer word. Tokens matter because they affect both the cost of a request and how much text fits in the context window.
2. What is a context window?
Model answer: It is the maximum amount of text the model can consider at one time, including instructions, chat history, files and its own answer. As a tester, I keep context focused by sharing only the relevant requirement or log section and starting a fresh chat for a new task.
3. Why does an LLM give different answers to the same prompt?
Model answer: The model chooses each token from a set of probabilities, and it often does not pick the single most likely option every time. Temperature controls that variation. Model updates can also change results over time, so I test AI behaviour against acceptance criteria instead of exact text.
4. What is temperature, and what value would you use to generate test data?
Model answer: Temperature controls how much randomness is used when choosing the next token. For structured test data I would use a low value for more consistent output, and I would still validate the data against the expected format.
5. What is a training cutoff, and why should a test engineer care?
Model answer: It is the date up to which the model’s training data was collected. A model may suggest outdated syntax or not know a recent framework release, so I check generated code against the current official documentation and run it before trusting it.
📝 Assignment
Repeat the three-chat experiment with a feature from your own project, using made-up details only. Write 5 lines on what changed between runs, what stayed the same, and which test scenarios a human had to add.
❓ FAQ
Does the AI remember my earlier conversations?
It depends on the product and your settings. Some assistants offer a memory feature, but by default you should assume a new chat starts with no knowledge of earlier ones.
Is a bigger context window always better?
Not always. A bigger window lets you share more, but irrelevant text can still distract the model. Focused context usually gives better test ideas.
🚀 Next Steps
Well done on finishing this lesson. Complete the assignment, then continue with AI Hallucinations in Software Testing: Why AI Output Must Be Checked.