Tokens and Context Windows
By the end of this lesson
Explain how tokens and context limits drive cost and behaviour.
A token is a chunk of text, usually a few characters. Common words are one token. Longer or unusual words break into several. Punctuation, spaces and line breaks are tokens too. "Reimbursement" is not one word to a model, and an employee number is not one number.
Tokens matter for two hard-edged reasons. You are billed per token, in and out. And every model has a context window — a maximum number of tokens it can consider at once — which is a limit, not a guideline.
import tiktoken
encoder = tiktoken.get_encoding("o200k_base")
extract = "Reimbursement for standard-class rail travel requires a receipt."
ids = encoder.encode(extract)
print(len(ids))
print([encoder.decode([i]) for i in ids])
# Rough size of a retrieved-context prompt
system_prompt = 320
chunks = 5 * 450
history = 1_200
reserved_for_answer = 500
print(system_prompt + chunks + history + reserved_for_answer)- tiktoken is the tokeniser library for OpenAI models. Decoding each id on its own shows you the actual pieces, and it is worth running once on your own content — seeing "Reimbursement" split apart makes the rest of this lesson concrete.
- Different model families tokenise differently, so a count from one tokeniser is an estimate for another. Use the encoding that matches the model you call.
- The second block is the calculation that matters in practice. Your prompt is not just the question. It is the system prompt, the conversation so far, the document extracts you retrieved, and the space you must leave for the reply.
- That reserved space is the part people forget. The window covers input and output together. Fill it with input and there is no room left to answer in.
- A rough English rule of thumb: around 4 characters per token, so roughly 750 tokens per 500 words. Use it for capacity planning and count properly before you rely on a number.
Everything competing for space in one request:
- System prompt
- Your standing instructions. Sent on every single call, so every sentence you add is paid for on every request forever. This is the cheapest place to find savings and the easiest to let sprawl.
- Conversation history
- Previous turns, resent in full each time because the model retains nothing between calls. A chat that feels like it remembers is a chat where you are re-uploading the transcript. Cost per turn climbs as the conversation grows.
- Retrieved context
- Document extracts you supplied for this question. Usually the largest and most variable part in a knowledge assistant, and the part you have most control over.
- The user's message
- Normally small, until someone pastes in a forty-page PDF and your careful budget disappears. Measure it rather than assuming.
- Room for the answer
- Output tokens come out of the same window. Reserve them deliberately, and set a maximum so a long reply cannot push you over.
Input and output tokens are not equivalent. Illustrative figures below — providers publish current rates and they change, so check before you build a forecast on them:
| Input tokens | Output tokens | |
|---|---|---|
| What they are | Everything you send | Everything the model produces |
| Price | The cheaper of the two | Commonly several times the input rate |
| Effect on latency | Modest — processed largely in parallel | Dominant — generated one token at a time |
| Where they come from | Prompt, history, retrieved extracts | The reply, including any reasoning the model is asked to show |
| Cheapest lever | Retrieve less, trim the system prompt, summarise history | Ask for brevity and cap the maximum |
A conversation eventually exceeds the window. It is not an edge case, it is arithmetic, so decide the policy before a user finds it:
Measure the prompt on every request
Count tokens for the assembled prompt before sending. Log the number. You cannot manage a budget you are not measuring, and the count is cheap to compute.
Drop the oldest turns
The simplest policy: keep the system prompt, keep the last few exchanges, discard the rest. Cheap and predictable. The cost is real — the assistant will forget the employee's grade if they mentioned it twenty turns ago.
Summarise instead of discarding
Replace older turns with a short running summary produced by a cheaper model. Keeps the thread of a long conversation at the price of an extra call and some lost detail. Never summarise anything you are required to retain — keep the full transcript in your own store.
Reset deliberately
For a question-answering assistant, most questions are independent. Starting a fresh context per question is often better than maintaining history nobody needs. Make that a decision rather than an accident.
Fail clearly when you cannot fit
If the user's own input exceeds what you can process, say so and say what to do instead. A context-length error surfaced as a generic failure is a support ticket waiting to happen.
Summary
- Tokens are sub-word pieces, not words, and you are billed per token in both directions
- Output tokens usually cost more than input tokens and dominate how long a reply takes to appear
- The context window is a hard limit covering system prompt, history, retrieved context and the answer
- Long conversations must eventually be truncated or summarised — decide the policy before a user finds the limit
- Extra context is never free: it costs money, adds latency, and can dilute the passage that mattered
Practice
Attempt each one before opening the solution. Getting it wrong first is how the idea sticks.
Try it yourself
Budget one request
Your model has a 128,000 token window. Your system prompt is about 400 tokens. Each retrieved chunk is about 500 tokens and you retrieve 6. You want to keep the last 10 conversation turns, averaging 150 tokens each, and reserve 800 tokens for the answer.
Work out the total. Then work out how many turns of history you could keep if a user pastes in a 90,000 token document.
Show solution
Base total: 400 system, 3,000 retrieved, 1,500 history, 800 reserved — 5,700 tokens. Comfortable, and worth noticing that comfortable is the normal case.
With a 90,000 token paste: 90,000 plus 400 plus 3,000 plus 800 reserved leaves roughly 33,800 for history, which is far more turns than you would want. The window is not the binding constraint here.
Cost is. That request is over sixteen times the input of the base case, every time it is sent. If the paste is resent with each follow-up turn, you are paying for those 90,000 tokens repeatedly.
This is the point of the exercise: fitting inside the window is the easy test to pass. The question to ask is whether every token you are sending is earning its place.
Knowledge check
Nothing is recorded and there is no score. The explanation appears either way.
Saved in this browser only.