RTP and media transport
RTP carries the audio a caller hears, in small UDP packets — typically twenty milliseconds of sound each, fifty a second. Nothing is retransmitted, because a packet that arrives late has missed the moment it belonged to. Loss is concealed by inventing plausible audio, which is kinder to human ears than to a speech model.
The packet, and why it is not TCP#
Each RTP packet carries a payload type saying which codec produced it, a sequence number, a timestamp describing where the sound belongs in time, and a synchronisation source identifier naming the stream. The receiver reassembles by timestamp, not by arrival order.
Reliability is deliberately absent. Asking for a lost packet again would take at least one round trip, by which time the twenty milliseconds it held are in the past and playing them would be worse than not. So RTP runs over UDP and drops what is lost, while a companion control protocol reports back on jitter, loss and round-trip time so each side can adapt.
Jitter, and the buffer that trades time for smoothness#
Packets sent at even intervals do not arrive at even intervals. That variation is jitter, and playing audio the instant each packet arrives would produce audible stutter. The fix is a jitter buffer: hold a small amount of audio, play it out steadily, and let late arrivals slot into place.
The buffer is bought with delay. Every millisecond held is a millisecond added to the conversation's round trip, and it is added in both directions. Adaptive buffers grow when the network is unstable, which means a poor connection lengthens the silence before a reply as well as degrading it — the two symptoms people report separately have one cause.
| Condition | What a person notices | What a speech model notices |
|---|---|---|
| Low, steady loss | Almost nothing — concealment works | Degraded recognition on exactly the syllables that were lost |
| Bursty loss | Words clipped out | Confident transcription of something that was never said |
| High jitter | Longer gaps, occasional robotic artefacts | Later input, so a later reply |
| Transcoding between codecs | Slightly duller audio | Measurable accuracy loss, added latency at each conversion |
Codecs and the quality ceiling#
The classic telephone codec is uncompressed 8 kHz audio: cheap to encode, predictable, and the reason a call sounds like a call. Wideband codecs exist and are excellent between two internet endpoints, but a path that ends at a handset is limited by its narrowest link, and each conversion between formats costs both quality and a little time.
This is the ceiling every phone-based speech system works under. It cannot be raised by choosing a better model, and it is why a voice agent that is flawless in a browser demonstration mishears names and numbers on a real call.
Does Connect use RTP directly?#
The audio on every call is RTP, and no part of Connect parses it. On the realtime path the media platform terminates the media and hands the worker audio frames through its SDK; on the browser softphone the media is WebRTC's encrypted profile, negotiated by the browser. Connect's own layers deal in turns, transcripts and events, not in packets.
One consequence of that separation is a measurement rule worth stating plainly. Latency here is measured on the wire, never from transcript rows. The worker's TurnClock times the caller's last word — the moment the user state leaves *speaking* — to the first audio back, when the agent state enters *speaking*, and posts a timing event for every reply. The first real call showed '8 ms' beside replies the caller had waited seconds for, because a transcript row is written when a sentence is complete, not when sound started.
On the turn-based carrier path, Connect's own spoken lines are synthesised ahead of the carrier: a sentence goes to a text-to-speech service, comes back as a small 8 kHz audio file, is stored content-addressed, and the carrier is handed a play instruction rather than asked to read text. If that ever fails, the carrier's own voice speaks — the line is never left silent.
Questions#
Would a better internet connection make the agent reply faster?
A little, and less than people hope. A stable connection lets the jitter buffer stay small, which removes tens of milliseconds. The dominant terms are the model's first token and the end-of-turn decision, measured in hundreds of milliseconds to seconds — see voice latency.
Why does packet loss hurt recognition more than it hurts me?
Because concealment is designed for ears. It fills a gap with something plausible from the surrounding audio, which your brain smooths over and a recognition model takes at face value. The result is not a gap in the transcript but a confident wrong word.
Is call audio encrypted?
Between a browser and the media server, yes — WebRTC requires it. Across the public telephone network the honest answer is that the path is out of your control and includes operators you have no relationship with, which is one reason a call is a poor place to read out a secret.