Voice activity detection
Voice activity detection answers a single question, thousands of times a minute: is this fragment of audio speech? It does not decide whether someone has finished talking — that is a different job. Connect runs it in two places at once, because the model provider's own detection misses speech starts that the host needs to see.
One question, asked constantly#
A detector looks at short frames of audio — tens of milliseconds each — and labels them speech or not. That is the whole contract. Everything built on top of it, from silence timers to interruption handling to endpointing, consumes that stream of labels.
- Gating what is sent onward. Audio that is not speech does not need to reach a model, and a model billed on audio tokens is billed for whatever audio it is given.
- Noticing that someone started talking. This is what makes interruption possible at all; the moment of speech onset is the trigger.
- Feeding silence timers. Hang-up watchers and 'are you still there' checks are counting frames labelled not-speech.
- The first stage of end-of-turn detection, which then has to decide something much harder — see end-of-turn detection.
How detectors differ#
| Kind | Strength | Weakness |
|---|---|---|
| Energy threshold | Almost free; trivially implemented | A television, a fan or a passing lorry is 'speech' |
| Spectral and statistical | Better at rejecting steady noise | Struggles with babble — noise that is other people talking |
| Small neural detectors | Reliable in noise, still cheap enough to run on every frame on a normal processor | A model to load and version, and behaviour that changes when it is updated |
Sensitivity is a genuine trade rather than a setting with a correct value. Turn it up and a single-word 'yes' starts a turn — which is exactly what you want from a caller answering a question. Turn it up further and a cough, a door and a colleague across the room all start turns too.
Where it runs, and why that decides what you can do#
A realtime model provider runs its own detection inside the session. You can also run one in your own process, on the audio as it arrives. These are not redundant, because they have different powers.
- The provider's detector
- Is the only one the model itself reacts to. It decides when the model considers a turn to have started and ended, and therefore when it generates.
- A host detector
- Sees the audio first and can act immediately — pause an outgoing line, start a clock, refuse to speak over somebody — without waiting for the provider to agree.
Does Connect use voice activity detection?#
Yes, and in both positions deliberately.
The session is configured with high start-of-speech sensitivity, so a one-word 'yes' is enough to start a turn. On a phone call, where callers answer in single syllables far more often than they do in a demonstration, the alternative loses half the confirmations in a conversation.
The host also keeps its own view of who is speaking, called Presence, assembled from every transcript chunk as it arrives. The reason is specific: the provider's user-state signal on the model path misses speech starts, and a system that trusted it would talk over people. Presence is what enforces the rule that nothing is said into the line while the caller is speaking, or within 2.5 seconds of the last time they were heard.
On the host turn-detection path, a small neural detector runs in-process alongside an audio end-of-turn model, and that path only works when the model's own detection is switched off — the authority question above, answered explicitly rather than left to chance.
Questions#
Is voice activity detection the same as end-of-turn detection?
No, and conflating them is the most common design error in voice agents. Detection labels audio as speech or silence. End-of-turn detection decides whether a silence means 'I have finished' or 'I am thinking of the next word', which requires understanding what was said.
Does a background television make a voice agent talk nonsense?
It can make it start turns it should not, which is worse than nonsense — the agent interrupts a caller who was listening. Noise-robust detection helps, and so does a rule that the agent does not speak while the caller is or has just been heard, which is what Presence enforces here.
Should I run my own detector if the provider already has one?
Only if you need to act on audio before the provider does — to hold a line silent, to time something honestly, or to make your own end-of-turn decision. If you do, decide which detector is authoritative and turn the other one off for that decision.