Speech-to-speech models
A speech-to-speech model hears the caller's audio and answers in audio directly, with no recogniser and no synthesiser in between. That removes two hops of delay and keeps the tone of the reply attached to its meaning. The cost is control: the words are decided as they are spoken, the transcript becomes a by-product rather than the input, and the voice is a property of the model rather than something you choose afterwards.
The chain it replaces#
The older arrangement is three components in a line. A recogniser turns the caller's audio into text; a language model reads that text and writes a reply; a synthesiser turns the reply into audio. Each stage has to finish before the next can start, so the caller's silence is the sum of three latencies plus the network between them.
- The caller speaks; audio arrives from the carrier leg.
- The recogniser turns it into text, but only once something has decided the turn ended.
- The text model reads that and writes a reply.
- The synthesiser turns the reply into audio.
- The first word finally plays down the line.
A speech-to-speech model collapses the middle three into one. It is also the only arrangement in which the reply can carry information the text never had: a caller who says "fine" flatly and a caller who says it warmly produce the same five characters, and a model that heard the audio can answer the difference.
What you give up#
- Exact wording
- You instruct a style, not a script. A sentence you need said the same way every time is not a thing this path guarantees, which is why the greeting is handled separately.
- The transcript as input
- Text is produced alongside the audio rather than ahead of it, so anything that reads the transcript is reacting after the caller has already heard the reply.
- Independent voice choice
- The voice comes with the model. Choosing a different voice is choosing a different model, or moving that piece of audio to a synthesiser.
- Cheap tokens
- Audio is metered at $3.00 per million in and $12.00 per million out, against $0.75 and $4.50 for text. A long call is priced on audio, and Connect prices every call on that basis before it is placed.
- Parameter freedom
- Settings that a text endpoint accepts can end a live audio session outright. Capping output tokens, or zeroing a thinking budget, has closed sessions on this path rather than shortening replies.
Does Connect use a speech-to-speech model?#
Used. The realtime voice path runs a Gemini Live speech-to-speech model, and it is the path a workspace opts into for live calls. Connect runs two phone engines, not one: a turn-based carrier path, and this realtime path. A workspace that has not opted in is on the turn-based engine and is not talking to a speech-to-speech model at all.
Text-to-speech has not disappeared. The greeting is synthesised with Google Text-to-Speech and played while the line is still connecting, so the first thing a caller hears is fixed wording that arrives without waiting for the model's first token. Everything after the greeting is the speech-to-speech model.
Conversational behaviour that only exists here#
Because the model is listening while it speaks, interruption is native rather than bolted on. Connect ignores apparent barge-in for the first 2 seconds of a reply, because a caller's "mm-hm" over the opening words is acknowledgement, not an interruption, and cutting the greeting in half is the failure that rule exists to prevent.
A watchdog sits above the model rather than inside it, with a default of 5.5 seconds. Its job is to notice that nothing is coming back and act; the reason it must sit above is that a nudge issued into a generation that is merely slow cancels the reply that was on its way, turning a pause into silence. Silence on an answered call is the worst outcome available, because the caller cannot tell it from a dead line.
Call length is bounded at 1,800 seconds. Long before that, cost metering will have priced the call in audio tokens against the workspace's budget, and a call the budget cannot cover is stopped rather than left to run.
Questions#
If the model produces text as well, why is the transcript late?
The audio is the primary output; the text is emitted alongside it and settles after the utterance rather than before. Anything that needs to act on what was said — a summary, a follow-up, an audit entry — is working from a record of a conversation that has already happened. Design for that rather than against it.
Can Connect say one exact sentence on a live call?
For the greeting, yes, because the greeting is synthesised rather than generated. Inside the conversation you get an instructed style and a constrained length, not a guaranteed script. Where exact wording is a legal or commercial requirement, the safe place for it is the greeting or a written channel.
Does removing the recogniser mean there is no end-of-turn decision?
No — something still has to decide the caller has stopped. The decision moves inside the model rather than disappearing, which is why turn detection is a separate topic and why hosted voice-activity detection is not automatically faster than the model's own.