Speech recognition
Speech recognition turns audio into text. On a turn-based voice path that text is the input a model reasons over, so recognition quality is answer quality. On Connect's realtime path there is no recogniser in front of the model: the transcript is produced beside the audio rather than ahead of it, which makes it a record of the call rather than the thing the call was decided from.
Two very different jobs with the same name#
Ask what depends on the text and the answer separates cleanly. In the older arrangement everything does: if the recogniser hears "can't" as "can", the model reasons over the opposite of what was said and answers confidently in the wrong direction, and nothing downstream can detect it. Recognition errors there are indistinguishable from the caller having said something else.
On a speech-to-speech path the model heard the audio. Ambiguity that a transcript flattens — hesitation, emphasis, a name half-swallowed — is still present when the reply is decided. The text produced alongside is for people and for records: the call summary, the audit trail, the follow-up somebody has to keep.
| Path | Text arrives | If it is wrong |
|---|---|---|
| Turn-based recognise → reason → synthesise | Before the reply is decided | The reply is wrong, and confidently so |
| Realtime speech-to-speech | Beside and after the audio | The record is wrong; the reply the caller heard was decided from audio |
Why it arrives late, and what to build around that#
Text settles after an utterance rather than during it. Anything that reads the transcript is therefore reacting to a conversation that has already moved on — by a beat at best, by a whole exchange at worst. This is not a defect to be tuned out; it is the shape of the path.
The practical consequence is a rule about what may depend on a transcript. A summary, a follow-up row, a decision-log entry and a relationship timeline are all after-the-fact and are safe. Anything that has to change what the caller hears next is not: by the time the words exist, they have been spoken.
Does Connect use speech recognition?#
Not as a separate stage on the realtime path. There is no standalone recogniser in front of the Gemini Live model — that is the point of a speech-to-speech architecture, and removing the hop is where a large part of the latency saving comes from.
A transcript is available, and it is production capability. call_transcript and call_summary are both available, produced through voice.py, and they are what a finished call leaves behind for both audiences. The turn-based carrier path derives its text from the carrier engine's own handling of the turn rather than from a recogniser Connect runs.
Reading a transcript honestly#
- Treat unclear as unclear. A transcript that shows a clean sentence where the caller mumbled has smoothed something. The reply the caller heard was decided from what was actually said.
- Names and numbers are the weak points. Account references, spellings and phone numbers are where a written record most often diverges from the call, and they are exactly the fields somebody will later act on.
- Silence is information. A gap in a transcript can mean a caller thinking, a caller gone, or a model that produced nothing — three very different incidents that look identical in text.
- Language switching survives poorly. A caller who moves between languages mid-sentence is intelligible to a model that heard it and awkward on the page.
Questions#
Can I search calls by what was said?
A transcript and a summary are attached to the call record, which lives alongside conversations and appears on the relationship timeline for the person it belongs to. Search the record; do not expect a live index of in-progress speech.
Why does the summary sometimes read better than the transcript?
Because a summary is written from the call by a model that had more than the text, while a transcript is a literal record with the disfluencies left in. When they disagree on a detail that matters — a date, an amount, a commitment — the safe move is to treat neither as settled and confirm it in writing on a channel that leaves evidence.
Does a better recogniser fix a wrong answer on a live call?
On a turn-based path, often yes, because the text is the input. On the realtime path there is no recogniser to improve; a wrong answer there is a model, instruction or Knowledge problem, and swapping a transcription component changes nothing about it.