# Voice latency

The gap before a voice agent answers is made of a few large terms and many small ones. On Connect's realtime engine the default model reaches first audio in 0.6 s, the best measured call had a 3.3 s median reply, and the review treats 2.5 s as the target and 4 s as slow. Some of that gap has no setting behind it at all.

- **Status:** Reference
- **Audience:** both, developer
- **Channels:** phone
- **In the app:** #/phone
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/docs/technology/voice-latency/

## A budget, not a number

Latency on a call is cumulative and it repeats. A term of half a second is not half a second of annoyance — it is half a second on each of twenty exchanges, which is what turns a technically fast system into one people describe as sluggish.

**Deciding the caller has finished** — Hundreds of milliseconds, deliberately spent. Connect's fastest end-of-speech setting waits 450 ms.
**The model's first token** — **0.6 s** on the current default model, measured on the host. On the older native-audio model it was **1.2–2.2 s** — the single largest difference available.
**The instructions the model must read first** — Measured directly: **7,500 characters gave a 1.2–1.8 s first token; 9,600 gave 2.3–3.4 s**. Prompt size is latency, not just cost.
**Speaking the reply** — Streams, so it overlaps the caller's listening rather than adding to the wait — provided the first audio came quickly.
**Network, jitter buffering and transcoding** — Tens of milliseconds each, and largely not yours to change.

Add the top three honestly and the floor becomes visible. The best measured call here had a **3.3 s median reply**, and that is the model's first token plus its end-of-turn decision — not a tuning failure and not something a settings panel removes.

## What a caller actually perceives

Two things matter more than the total, and both are exploitable.

1. **When sound starts, not when the sentence finishes.** A reply that begins quickly and runs for six seconds feels faster than one that begins after three and runs for two. This is why first-audio timing is the number worth optimising.
2. **The pickup moment.** A greeting synthesised in advance plays the instant the caller picks up. Warming greetings in each idle worker process took the inbound greeting from **3.0 s to nothing at all** — the largest single improvement in this whole area, and it happens before the model is involved.

Reply length pulls in the same direction. After the instructions were changed to ask for one thought and one question under twenty-five words, the median words per reply fell from **23–40 to 19**, re-introductions went to zero, stacked questions went to zero, and the reply median settled at 2.9 s **with no latency regression**. Shorter answers did not cost speed; they improved the conversation independently.

> **Note** The warm-up itself had a failure worth knowing about. Every restart warmed nothing for a period, because each idle process asked the synthesiser for the same sentence with the same key in the same millisecond, one was refused by a per-minute quota, and the greeting was then given up for the life of that process. Processes are staggered now, refusals that can pass are retried with backoff, and a refusal is counted separately from a failure.

## Measuring it without lying to yourself

The rule that matters more than any threshold: **latency is measured on the wire, never from transcript rows.** The worker's clock times the caller's last word — the moment their speaking state ends — to the first audio back, and posts a timing event for every reply.

Measuring from transcripts produces confident nonsense. The first real call on this engine showed '8 ms' beside replies the caller had waited seconds for, because a transcript row is written when a sentence is complete, not when sound began. A dashboard built on that would have reported a system faster than anything physically possible while callers were sitting in silence.

> **Careful** A related trap is the impatient recovery timer. The watchdog that prompts a stalled model was originally set below the model's normal response time; because a prompt issued during an in-flight generation cancels it, the timer designed to rescue slow turns was creating them. It sits at 5.5 s for that reason.

## Does Connect use a latency budget, and what does it do with it?

Yes. Every realtime call carries per-reply timings, and the review that runs when a call finishes turns them into findings: replies slower than the **4 s** threshold, long gaps, ignored interruptions, false stops, repetition and unheard callers, each paired with the setting that would address it. The target it judges against is **2.5 s**.

The part worth borrowing is the honesty rule. When the model's own first token accounts for **60%** of the wait, the finding says so and offers no control to turn. Those model-limited findings are listed and counted but do not lower the call's score, because scoring a call down for something no operator can change teaches nothing and trains people to ignore the score.

The settings that were actually resolved for the call are snapshotted on the call record, so a review months later judges what ran rather than what is configured today.

## Questions

### Can a voice agent reply as fast as a person?

Not on this stack today. The floor is the model's first token plus the end-of-turn decision, and on the best measured call that produced a 3.3 s median. Anyone quoting one second for a phone agent is either measuring something else or measuring from a transcript.

### What is the cheapest thing to change?

The instruction block. Nine and a half thousand characters cost roughly a second more first token than seven and a half thousand, and most of that difference is knowledge and memory that could be tighter. The budgets exist for that reason: four facts and 2,000 characters of knowledge, 700 for memory, 600 for the contact block.

### Does a faster model always win?

No — it is a trade. The fastest default model here refuses mid-call instruction updates, so a supervisor's guidance and mid-call notes cannot reach it and are recorded as undelivered. When steering a live call matters more than a second per reply, the slower model is the right choice, and that is a per-workspace decision rather than a default.

## Related

- [Measuring conversational latency correctly](https://connectbyjbrh.com/research/measuring-voice-latency/)
- [What every character of a prompt costs on a live call](https://connectbyjbrh.com/research/prompt-size-and-first-token/)
- [End-of-turn detection](https://connectbyjbrh.com/docs/technology/turn-detection/)
- [Realtime voice agents](https://connectbyjbrh.com/docs/technology/realtime-voice/)
- [Findings a setting cannot fix](https://connectbyjbrh.com/research/model-limited-findings/)

## What this page is based on

- Source pack: phone and voice — `docs-source/sources/PHONE.md` §6, §7, §8, §10 and §12 (TurnClock and the transcript-row error, target and slow thresholds, the 60% model-bound share, instruction-size measurements, greeting warm-up from 3.0 s to none and its quota failure, the gap audit's reply-length result, the 3.3 s floor)
- Connect capability registry (`docs-source/facts.py`) — `MEASURED` for every figure quoted on this page