Context window
A context window is the maximum amount of text a model can consider in a single call, counted in tokens. Everything the model is meant to know for that call has to be inside it — instructions, history, retrieved material, the current message — and it is re-sent every turn, because the model retains nothing between calls. Window size is a capacity limit, not a memory.
What is inside the window on a typical turn#
- Instructions. Who the model is acting as, what it may and may not do, how to write. Constant across turns and therefore paid for on every one.
- Retrieved material. Whatever grounding step ran: the relevant Knowledge, the facts that apply, the record being discussed.
- Memory. What is known about this business, this channel, this person.
- History. Earlier turns of this conversation, in full or summarised.
- The message itself, which is usually the smallest part.
The proportions surprise people. On a live call the instruction block and the grounding material together are routinely twenty times the size of what the caller just said, which is why prompt size is a latency decision rather than a matter of taste.
Why a bigger window is not a bigger memory#
Three separate things get conflated, and the difference decides your architecture.
- Window
- How much can be sent at once. A hard technical limit set by the model.
- Memory
- What survives between conversations, stored outside the model in records that can be read, corrected and forgotten. See Memory in Connect.
- Attention quality
- How reliably the model uses what was sent. This does not scale with window size; material in the middle of a very long context is used less consistently than material at either end.
So filling a large window with a whole mailbox is not the same as remembering a customer. It is expensive, it is slower, it degrades precision, and it vanishes the moment the call ends. Structured records solve the problem the big window appears to solve — and they can be queried, audited and corrected, which a prompt cannot.
Does Connect use a context window?#
Used, necessarily — every model call has one — and the notable part is that Connect treats it as a budget to be spent rather than a space to be filled. The grounding budgets on the realtime voice path are fixed and small on purpose.
| Block | Budget |
|---|---|
| Knowledge | 2,000 characters, at most 4 facts |
| Memory | 700 characters |
| Contact block | 600 characters |
| A human voice profile's prompt | about 2,500 characters |
Those numbers are measured, and the measurement they come from is a latency one. An instruction block of about 7,500 characters produced a first token in 1.2–1.8 seconds; at about 9,600 characters the same model took 2.3–3.4 seconds. On a phone call that difference is the gap between a reply and a silence the caller starts talking into. What every character of a prompt costs on a live call records the run.
On the mail side the constraint is cost and precision rather than latency, and the approach is the same: retrieve what applies to this message instead of sending everything and hoping the model picks correctly.
Running out of room#
Cut history before you cut grounding.
Result Older turns are usually the least load-bearing content in the window; the facts an answer must not contradict are the most.
Summarise rather than truncate.
Result A hard truncation removes the beginning of a conversation, which is where the customer usually said what they wanted. See context compression.
Move durable knowledge out of the prompt entirely.
Result Anything true across conversations belongs in Knowledge or Memory, retrieved when relevant, not carried in every call.
Questions#
Does a longer context window remove the need for retrieval?
No. It changes the ceiling, not the economics or the precision. Sending an entire knowledge base on every turn costs on every turn, slows the first token, and gives the model more chances to ground an answer in the wrong passage. Retrieval stays the cheaper and more accurate route at any window size.
Does Connect remember a conversation after it ends?
What Connect keeps lives in records — the canonical thread, the relationship, the memory tiers — not in a model's context. The next call starts a fresh window and is given what applies. That is why a memory can be read, corrected and deleted by a person, which nothing inside a model's context ever could be.
Why are the voice budgets so much smaller than the mail ones?
Because a caller is waiting. Characters in the instruction block are paid for in first-token time on every single reply, and the measured curve above steepens quickly. Mail can afford context that a live call cannot.