Voice latency
The gap before a voice agent answers is made of a few large terms and many small ones. On Connect's realtime engine the default model reaches first audio in 0.6 s, the best measured call had a 3.3 s median reply, and the review treats 2.5 s as the target and 4 s as slow. Some of that gap has no setting behind it at all.
A budget, not a number#
Latency on a call is cumulative and it repeats. A term of half a second is not half a second of annoyance — it is half a second on each of twenty exchanges, which is what turns a technically fast system into one people describe as sluggish.
- Deciding the caller has finished
- Hundreds of milliseconds, deliberately spent. Connect's fastest end-of-speech setting waits 450 ms.
- The model's first token
- 0.6 s on the current default model, measured on the host. On the older native-audio model it was 1.2–2.2 s — the single largest difference available.
- The instructions the model must read first
- Measured directly: 7,500 characters gave a 1.2–1.8 s first token; 9,600 gave 2.3–3.4 s. Prompt size is latency, not just cost.
- Speaking the reply
- Streams, so it overlaps the caller's listening rather than adding to the wait — provided the first audio came quickly.
- Network, jitter buffering and transcoding
- Tens of milliseconds each, and largely not yours to change.
Add the top three honestly and the floor becomes visible. The best measured call here had a 3.3 s median reply, and that is the model's first token plus its end-of-turn decision — not a tuning failure and not something a settings panel removes.
What a caller actually perceives#
Two things matter more than the total, and both are exploitable.
- When sound starts, not when the sentence finishes. A reply that begins quickly and runs for six seconds feels faster than one that begins after three and runs for two. This is why first-audio timing is the number worth optimising.
- The pickup moment. A greeting synthesised in advance plays the instant the caller picks up. Warming greetings in each idle worker process took the inbound greeting from 3.0 s to nothing at all — the largest single improvement in this whole area, and it happens before the model is involved.
Reply length pulls in the same direction. After the instructions were changed to ask for one thought and one question under twenty-five words, the median words per reply fell from 23–40 to 19, re-introductions went to zero, stacked questions went to zero, and the reply median settled at 2.9 s with no latency regression. Shorter answers did not cost speed; they improved the conversation independently.
Measuring it without lying to yourself#
The rule that matters more than any threshold: latency is measured on the wire, never from transcript rows. The worker's clock times the caller's last word — the moment their speaking state ends — to the first audio back, and posts a timing event for every reply.
Measuring from transcripts produces confident nonsense. The first real call on this engine showed '8 ms' beside replies the caller had waited seconds for, because a transcript row is written when a sentence is complete, not when sound began. A dashboard built on that would have reported a system faster than anything physically possible while callers were sitting in silence.
Does Connect use a latency budget, and what does it do with it?#
Yes. Every realtime call carries per-reply timings, and the review that runs when a call finishes turns them into findings: replies slower than the 4 s threshold, long gaps, ignored interruptions, false stops, repetition and unheard callers, each paired with the setting that would address it. The target it judges against is 2.5 s.
The part worth borrowing is the honesty rule. When the model's own first token accounts for 60% of the wait, the finding says so and offers no control to turn. Those model-limited findings are listed and counted but do not lower the call's score, because scoring a call down for something no operator can change teaches nothing and trains people to ignore the score.
The settings that were actually resolved for the call are snapshotted on the call record, so a review months later judges what ran rather than what is configured today.
Questions#
Can a voice agent reply as fast as a person?
Not on this stack today. The floor is the model's first token plus the end-of-turn decision, and on the best measured call that produced a 3.3 s median. Anyone quoting one second for a phone agent is either measuring something else or measuring from a transcript.
What is the cheapest thing to change?
The instruction block. Nine and a half thousand characters cost roughly a second more first token than seven and a half thousand, and most of that difference is knowledge and memory that could be tighter. The budgets exist for that reason: four facts and 2,000 characters of knowledge, 700 for memory, 600 for the contact block.
Does a faster model always win?
No — it is a trade. The fastest default model here refuses mid-call instruction updates, so a supervisor's guidance and mid-call notes cannot reach it and are recorded as undelivered. When steering a live call matters more than a second per reply, the slower model is the right choice, and that is a per-workspace decision rather than a default.