LiveKit Agents and the voice worker
LiveKit is the realtime media platform under Connect's live phone and WhatsApp calls. A call from the carrier arrives over SIP into a LiveKit room. A dispatch rule starts Connect's connect-voice worker in that room, and the worker, built on the LiveKit Agents framework, connects the caller's audio to a realtime speech-model session. The worker knows no business logic. It asks the application for everything.
What it is#
LiveKit is an open-source WebRTC platform from LiveKit, Inc. It has a media server (an SFU), client SDKs, a SIP service that bridges phone calls into rooms, and the Agents framework for writing programs that join a room as a participant. LiveKit Cloud is the company's hosted service.
The Agents framework handles the hard real-time parts: audio frames, voice activity, interruptions, and plugins for speech and model providers. Connect uses a model-provider plugin for the realtime speech model, the Silero plugin for host-side voice activity detection, and the noise-cancellation plugin.
- Category
- Realtime media and voice-agent framework
- Versions in Connect
livekit==1.1.17,livekit-api==1.2.1,livekit-agents==1.8.0, the model-provider plugin at 1.8.0,livekit-plugins-silero==1.8.0,livekit-plugins-noise-cancellation==0.3.0- Project
- A LiveKit project configured on the host (
LIVEKIT_URLand its key pair) - Official site
- docs.livekit.io
Where Connect uses it#
| Place in Connect | What it does there |
|---|---|
backend/voice_worker/agent.py | The worker: joins the room, runs the realtime session, posts each sentence and event to the application |
backend/app/voice_engine.py | Renders the call brief once per call and receives the worker's posts |
backend/app/whatsapp_calling.py | WhatsApp calls through the same engine, with an output gate on that leg only |
backend/voice_worker/audio_gate.py | The noise gate on the agent's voice for WhatsApp legs |
How it connects#
- The carrier sends the call to the project's SIP host
- LiveKit SIP creates a room, and a dispatch rule starts the worker
- The worker fetches the brief from the application on the loopback port
- Caller audio passes noise filtering, then the realtime model session
- At the end the worker posts
/engine/end, and the application writes the summary, memory and follow-up
Benefits and limits#
What it gives Connect:
- Barge-in, automatic language and one consistent voice, because the model listens and speaks at once.
- A thin worker: business rules stay in the application, where they are tested.
- Exact pins, proven together on a real call, so an upgrade is a deliberate, measured change.
Where it stops, and what Connect does about it:
- The calling quality is locked. Engine settings, filters, turn-taking and the tool set are pinned by a suite, and changing them needs the owner's approval and a measured call.
- The enhanced noise-cancellation plugin requires LiveKit Cloud.
- Realtime sessions are billed by audio duration on both LiveKit and the model provider. A worker never holds a session past its maximum duration.
Security and privacy#
- The worker talks to the application over loopback with its own token. The endpoint answers 503 when the token is unset, never open.
- LiveKit credentials live in the host's environment file.
- Recordings and transcripts follow the workspace's settings. A transcript is evidence, not a clock (call transcripts).
Licence, cost and versions#
| Question | Answer |
|---|---|
| Licence | LiveKit server, SDKs and Agents: Apache-2.0. The hosted service and its enhanced noise-cancellation models are commercial |
| Cost | LiveKit Cloud usage per its pricing page. The model's audio is billed separately and metered by Connect per call |
| Version | Exact pins in requirements-voice.txt |
| Compatibility | Installed into the same venv as the application on the host; the web application never imports the realtime SDK |
Troubleshooting#
- The call connects but nobody speaks
- The worker was not dispatched or failed to start its model session. Check
journalctl -u connect-voiceand the call's engine events. - The caller hears the agent's own echo or hiss on WhatsApp
- The WhatsApp output gate. It is locked: report it with the call, do not tune it ad hoc.
- Calls end at a fixed length
- The session reached
max_seconds. That is the cost guard.
Official references#
Questions#
Does every call use LiveKit?
Calls on lines set to the realtime engine do. A line can also use the carrier's turn-based path, where the carrier plays Connect's written reply. Both end in the same call record.
Why is the voice worker's SDK not in the main requirements?
So a broken audio stack cannot stop the web application, and the mail sync, from starting. The two run as separate services from one venv.