The realtime voice engine

What happens to a call on this path#

1. TRIGGER — a call arrives on a number whose workspace has voice.engine = "realtime"
2. USER / EXTERNAL EVENT — the carrier opens a SIP leg into the project's inbound trunk
3. AUTH / WORKSPACE RESOLUTION — the number resolves to one workspace, and the handler enters it before reading anything
4. INGEST / REQUEST — a dispatch rule spawns the connect-voice worker on the room; admission refuses not_ready if no worker has checked in
5. CANONICAL RECORD — a Call row exists from acceptance, with the resolved voice settings snapshotted onto quality
6. CLASSIFICATION / RESEARCH / REASONING — a realtime model listens and speaks on the audio stream; the greeting is played from audio synthesised during the ring
7. KNOWLEDGE + MEMORY + RULES — voice_engine.brief() renders persona, line purpose, contact facts, standing instructions, knowledge and the hard rules once per call
8. AUTONOMY / APPROVAL — outbound goes through comms_providers.dispatch_call with a placer, so consent, blocks, autonomy and budget gate it exactly as a carrier call
9. ACTION / PROVIDER — the worker speaks; every sentence is posted to /api/voice/engine/spoken and the guardrails are re-checked after the fact
10. RESULT — /engine/end runs the same voice.end_call the turn-based path runs
11. RELATIONSHIP / TIMELINE / MEMORY — summary, lead, memory and follow-up run in the deferred digest
12. AUDIT / USAGE / NEEDS YOU — model usage lands in costs; line health escalates into Needs You and drains as its cause clears

The stages that differ from the carrier path are the middle ones. Everything from the canonical record onwards is shared code, which is why changing engines does not change what a call leaves behind.

Turning it on#

1. Set voice.engine = "realtime" on the workspace.

   Result Calls on the workspace's lines are held by the engine rather than by the carrier's turn loop.

2. At the carrier, link the number to an inbound trunk. This is console work and cannot be done through an API on the live provider.

   Result Calls arrive as a SIP leg rather than as a series of webhooks.

3. Set the trunk's origination URI to the LiveKit project's SIP host as host:5060, with no sip: prefix.

   Result The leg reaches the project. The host is the project-id subdomain — LIVEKIT_SIP_HOST — not the hostname in the server URL used elsewhere.

4. Confirm a worker is checked in before testing.

   Result Admission refuses before a row exists when nothing has heartbeated for 90 seconds, so a test against an empty fleet tells you nothing about the configuration.

What the worker knows, and what it must not#

It knows the brief

Persona, the line's purpose, the contact's facts and standing instructions, knowledge, the hard rules and the supervisor's guidance — rendered once per call by voice_engine.brief().

It does not know the business

No queries, no rules engine, no records. If a fact is not in the brief, the voice does not have it, and that is deliberate: the worker is the mouth, not the mind.

It reports rather than decides

Sentences to /engine/spoken, events to /engine/event, the ending to /engine/end. Guardrails are applied on Connect's side, after the fact, where the authority lives.

It cannot open its own door

The worker's endpoint is authenticated by MAYA_VOICE_ENGINE_TOKEN over loopback. With no token set it answers 503 — closed, never open.

Keeping the brief the only channel has a cost worth naming: prompt size is latency. Instructions of 7,500 characters measured a 1.2–1.8 s first token; 9,600 characters measured 2.3–3.4 s. That is why the budgets are fixed — four facts and 2,000 characters of knowledge, 700 for memory, 600 for the contact block — rather than left to grow.

What this engine buys, and what it costs#

It buys interruption and speed. A caller can speak over the voice and be heard, a one-word answer starts a turn, and the default model reaches first audio in about 0.6 s on the host — against 1.2–2.2 s for the 2.5-generation native-audio model.

It costs money and flexibility. A live call is billed on audio tokens — four times the same model's text rate on input and roughly 2.7 times on output — and a Live session is re-billed for its whole context every turn, with production calls running 92% carried context by input token. The default model's session is also immutable, so mid-call notes and supervisor guidance cannot reach it: the worker records the attempt as undelivered rather than pretending it landed. Choosing the 2.5 model buys mid-call steering back at roughly a second per reply.

Questions#

Related#

• The carrier turn-based engine
• The voice worker
• Capacity and admission control
• What a call costs
• What a setting cannot change
• Designing telephony that does not name its provider