Connect by JBRH Open Connect

LiveKit Agents and the voice worker

LiveKit is the realtime media platform under Connect's live phone and WhatsApp calls. A call from the carrier arrives over SIP into a LiveKit room. A dispatch rule starts Connect's connect-voice worker in that room, and the worker, built on the LiveKit Agents framework, connects the caller's audio to a realtime speech-model session. The worker knows no business logic. It asks the application for everything.

Status
Reference What this means
Audience
both, developer
Last verified
Product version
6.3.2

What it is#

LiveKit is an open-source WebRTC platform from LiveKit, Inc. It has a media server (an SFU), client SDKs, a SIP service that bridges phone calls into rooms, and the Agents framework for writing programs that join a room as a participant. LiveKit Cloud is the company's hosted service.

The Agents framework handles the hard real-time parts: audio frames, voice activity, interruptions, and plugins for speech and model providers. Connect uses a model-provider plugin for the realtime speech model, the Silero plugin for host-side voice activity detection, and the noise-cancellation plugin.

Category
Realtime media and voice-agent framework
Versions in Connect
livekit==1.1.17, livekit-api==1.2.1, livekit-agents==1.8.0, the model-provider plugin at 1.8.0, livekit-plugins-silero==1.8.0, livekit-plugins-noise-cancellation==0.3.0
Project
A LiveKit project configured on the host (LIVEKIT_URL and its key pair)
Official site
docs.livekit.io

Where Connect uses it#

Place in ConnectWhat it does there
backend/voice_worker/agent.pyThe worker: joins the room, runs the realtime session, posts each sentence and event to the application
backend/app/voice_engine.pyRenders the call brief once per call and receives the worker's posts
backend/app/whatsapp_calling.pyWhatsApp calls through the same engine, with an output gate on that leg only
backend/voice_worker/audio_gate.pyThe noise gate on the agent's voice for WhatsApp legs

How it connects#

  1. The carrier sends the call to the project's SIP host
  2. LiveKit SIP creates a room, and a dispatch rule starts the worker
  3. The worker fetches the brief from the application on the loopback port
  4. Caller audio passes noise filtering, then the realtime model session
  5. At the end the worker posts /engine/end, and the application writes the summary, memory and follow-up

Benefits and limits#

What it gives Connect:

  • Barge-in, automatic language and one consistent voice, because the model listens and speaks at once.
  • A thin worker: business rules stay in the application, where they are tested.
  • Exact pins, proven together on a real call, so an upgrade is a deliberate, measured change.

Where it stops, and what Connect does about it:

  • The calling quality is locked. Engine settings, filters, turn-taking and the tool set are pinned by a suite, and changing them needs the owner's approval and a measured call.
  • The enhanced noise-cancellation plugin requires LiveKit Cloud.
  • Realtime sessions are billed by audio duration on both LiveKit and the model provider. A worker never holds a session past its maximum duration.

Security and privacy#

  • The worker talks to the application over loopback with its own token. The endpoint answers 503 when the token is unset, never open.
  • LiveKit credentials live in the host's environment file.
  • Recordings and transcripts follow the workspace's settings. A transcript is evidence, not a clock (call transcripts).

Licence, cost and versions#

QuestionAnswer
LicenceLiveKit server, SDKs and Agents: Apache-2.0. The hosted service and its enhanced noise-cancellation models are commercial
CostLiveKit Cloud usage per its pricing page. The model's audio is billed separately and metered by Connect per call
VersionExact pins in requirements-voice.txt
CompatibilityInstalled into the same venv as the application on the host; the web application never imports the realtime SDK

Troubleshooting#

The call connects but nobody speaks
The worker was not dispatched or failed to start its model session. Check journalctl -u connect-voice and the call's engine events.
The caller hears the agent's own echo or hiss on WhatsApp
The WhatsApp output gate. It is locked: report it with the call, do not tune it ad hoc.
Calls end at a fixed length
The session reached max_seconds. That is the cost guard.

Official references#

Questions#

Does every call use LiveKit?

Calls on lines set to the realtime engine do. A line can also use the carrier's turn-based path, where the carrier plays Connect's written reply. Both end in the same call record.

Why is the voice worker's SDK not in the main requirements?

So a broken audio stack cannot stop the web application, and the mail sync, from starting. The two run as separate services from one venv.