Realtime voice agents
A realtime voice agent holds a call directly: a speech-to-speech model listens and speaks at once, with no transcription-then-generation-then-speech pipeline in the middle. Connect runs one in production. The architecture that matters is not the model — it is what surrounds it, because everything that fails on a live call fails outside the model.
Two shapes, and why the older one is still reasonable#
| Classic pipeline | Realtime speech-to-speech | |
|---|---|---|
| Steps | Recognise → generate text → synthesise | One model hears audio and emits audio |
| Latency | The sum of three stages, each waiting for the last | One first-token wait |
| Interruption | Awkward: the pipeline is committed once synthesis starts | Native: the model can be cut off mid-utterance |
| What you can inspect | A clean text transcript at a defined boundary | Transcripts are a by-product, arriving when a sentence completes |
| Control | Every stage is yours to constrain | Behaviour is instructions plus session parameters, and some things simply cannot be set |
Connect runs both. The turn-based carrier path — a webhook hears a sentence, Connect writes a reply, the carrier reads it out — is still there, and a workspace opts into the realtime engine. The realtime path is faster and more natural; the turn-based path is easier to reason about and does not depend on a worker fleet being healthy.
The parts, and what each is responsible for#
- CARRIER — the public-network leg, and the number
- INBOUND TRUNK — delivers the call to the media platform
- DISPATCH RULE — spawns the voice worker on the call's room
- WORKER — runs the session, holds the clock, posts events
- BRIEF — persona, line purpose, contact facts, standing instructions, knowledge, hard rules, supervisor guidance, rendered once per call
- MODEL — a realtime speech-to-speech session
- ENGINE API — spoken lines, events, and the end of the call
- END OF CALL — summary, lead, memory, follow-up, and the cost booked
The division that makes this maintainable: the worker knows no business logic. It renders nothing, decides nothing about a customer, and holds no rules. It posts every sentence the voice says to the engine, where guardrails are re-checked after the fact; it posts events; and when the call ends the engine runs exactly the same end-of-call path a carrier call runs — summary, lead capture, memory, follow-up — plus the model's usage into the cost ledger.
Outbound calls go through the same dispatch function used everywhere else, carrying a recorded placer, so memory blocks, consent, autonomy and budget gate a realtime call exactly as they gate a carrier one. The worker's own door is authenticated by a token and answers 503 when that token is unset — it is never open.
The failure surface#
- The greeting cannot be synthesised
- The call ends deliberately, attributed to the engine, rather than leaving the caller in silence. This rule exists because five inbound calls once sat silent and were recorded as 'the caller hung up before speaking' — a fault that looked like disinterest.
- The model session refuses to start
- Opened once more on the same model, then once on the fallback model, before the call is given up. The restart is written to the record.
- A request the call cannot lose fails
- Arrival, brief and end are retried through a restart or a proxy error. A
4xxis an answer and is never retried — retrying a refusal is how one bad call becomes many. - A dial times out at the carrier
- Dialled once more. A leg the carrier answered that never joins the room is written off after a timeout rather than waited on indefinitely.
- The line is switched off or closed
- The caller hears a spoken closed-line message in the business's own name, and the call is recorded with an outcome saying which. Earlier this was a bare refusal that dropped the call with no record at all.
- The call was never a conversation
- It gets its own vocabulary: rang and nobody answered, versus never rang because the dial, the dispatch or the worker failed. Neither creates a lead.
Endings are attributed rather than assumed. Every call names who ended it — caller, model, owner, engine, carrier or agent — because before that existed, everything that was not the caller was blamed on the agent.
Does Connect use realtime voice, and what does it cost?#
Yes, in production, as an engine a workspace selects. The honest performance picture: the default model reaches first audio in 0.6 s measured on the host, against 1.2–2.2 s for the older native-audio model, and the best measured call had a 3.3 s median reply. That median is the floor of the model's first token plus its end-of-turn decision, not a tuning failure.
Cost behaves unlike text. A live call is billed on audio tokens, several times the same model's text rate on input and output, so metering is split by modality rather than totalled. More importantly, a live session is re-billed for its entire context on every turn — production calls are 92% carried context by input token — which is why context compression is sized rather than left at the model's whole context window, and why a long call costs disproportionately more than a short one.
Two things a realtime agent here does not do: calls are not recorded on the live carrier, and a completed warm handover to a colleague is not available — an escalation queues, and a supervisor can act on the live call, but the handover itself depends on a provider capability that is not enabled. A call also has a ceiling of thirty minutes.
Questions#
Is a realtime agent always better than the pipeline approach?
Not always. It is faster and interrupts naturally. It also gives you less control: some models refuse mid-call instruction updates entirely, and behaviour you would implement in code on a pipeline has to be requested in instructions and may simply not happen. Where determinism matters more than a second per reply, the turn-based path is defensible.
What happens to a call when a deployment goes out?
Calls in progress are given a drain period of 180 seconds to finish, and the service unit allows 210 before it is killed. Sessions also survive the provider's own reconnection, carrying the conversation, so the model's session limit is not the call's limit.
Where do the transcripts come from if the model works in audio?
The provider emits them alongside the audio, which makes them useful for review and useless for timing — a row is written when a sentence is complete. Anything measuring how long a caller waited must be measured on the wire instead, which is what the worker's own clock does.