# The realtime voice engine

On the realtime engine Connect holds the call itself: audio goes to a model that listens and speaks at once, rather than to a carrier that transcribes a sentence and reads a reply back. A workspace turns it on by setting `voice.engine = "realtime"` and linking the number at the carrier to an inbound trunk. The worker carries no business logic — everything it knows arrives in one brief.

- **Status:** Available
- **Audience:** both
- **Channels:** phone
- **In the app:** #/calls, #/phone-advanced
- **Last verified:** 2026-09-10
- **Canonical:** https://connectbyjbrh.com/docs/phone/realtime-engine/

## What happens to a call on this path

1. TRIGGER — a call arrives on a number whose workspace has `voice.engine = "realtime"`
2. USER / EXTERNAL EVENT — the carrier opens a SIP leg into the project's inbound trunk
3. AUTH / WORKSPACE RESOLUTION — the number resolves to one workspace, and the handler enters it before reading anything
4. INGEST / REQUEST — a dispatch rule spawns the `connect-voice` worker on the room; admission refuses `not_ready` if no worker has checked in
5. CANONICAL RECORD — a `Call` row exists from acceptance, with the resolved voice settings snapshotted onto `quality`
6. CLASSIFICATION / RESEARCH / REASONING — a realtime model listens and speaks on the audio stream; the greeting is played from audio synthesised during the ring
7. KNOWLEDGE + MEMORY + RULES — `voice_engine.brief()` renders persona, line purpose, contact facts, standing instructions, knowledge and the hard rules once per call
8. AUTONOMY / APPROVAL — outbound goes through `comms_providers.dispatch_call` with a `placer`, so consent, blocks, autonomy and budget gate it exactly as a carrier call
9. ACTION / PROVIDER — the worker speaks; every sentence is posted to `/api/voice/engine/spoken` and the guardrails are re-checked after the fact
10. RESULT — `/engine/end` runs the same `voice.end_call` the turn-based path runs
11. RELATIONSHIP / TIMELINE / MEMORY — summary, lead, memory and follow-up run in the deferred digest
12. AUDIT / USAGE / NEEDS YOU — model usage lands in `costs`; line health escalates into Needs You and drains as its cause clears

The stages that differ from the carrier path are the middle ones. Everything from the canonical record onwards is shared code, which is why changing engines does not change what a call leaves behind.

## Turning it on

1. Set `voice.engine = "realtime"` on the workspace.
   - Result: Calls on the workspace's lines are held by the engine rather than by the carrier's turn loop.
2. At the carrier, link the number to an inbound trunk. This is console work and cannot be done through an API on the live provider.
   - Result: Calls arrive as a SIP leg rather than as a series of webhooks.
3. Set the trunk's origination URI to the LiveKit project's SIP host as `host:5060`, with **no** `sip:` prefix.
   - Result: The leg reaches the project. The host is the project-id subdomain — `LIVEKIT_SIP_HOST` — not the hostname in the server URL used elsewhere.
4. Confirm a worker is checked in before testing.
   - Result: Admission refuses before a row exists when nothing has heartbeated for 90 seconds, so a test against an empty fleet tells you nothing about the configuration.

> **Note** Test the loop with the carrier's own outbound call API rather than by dialling from the project back into itself. A hairpin test exercises a path the real call never takes and passes when the real one would fail.

## What the worker knows, and what it must not

**It knows the brief** — Persona, the line's purpose, the contact's facts and standing instructions, knowledge, the hard rules and the supervisor's guidance — rendered once per call by `voice_engine.brief()`.
**It does not know the business** — No queries, no rules engine, no records. If a fact is not in the brief, the voice does not have it, and that is deliberate: the worker is the mouth, not the mind.
**It reports rather than decides** — Sentences to `/engine/spoken`, events to `/engine/event`, the ending to `/engine/end`. Guardrails are applied on Connect's side, after the fact, where the authority lives.
**It cannot open its own door** — The worker's endpoint is authenticated by `MAYA_VOICE_ENGINE_TOKEN` over loopback. With no token set it answers 503 — closed, never open.

Keeping the brief the only channel has a cost worth naming: prompt size is latency. Instructions of 7,500 characters measured a 1.2–1.8 s first token; 9,600 characters measured 2.3–3.4 s. That is why the budgets are fixed — **four facts and 2,000 characters of knowledge, 700 for memory, 600 for the contact block** — rather than left to grow.

## What this engine buys, and what it costs

It buys interruption and speed. A caller can speak over the voice and be heard, a one-word answer starts a turn, and the default model reaches first audio in about **0.6 s** on the host — against 1.2–2.2 s for the 2.5-generation native-audio model.

It costs money and flexibility. A live call is billed on audio tokens — four times the same model's text rate on input and roughly 2.7 times on output — and a Live session is re-billed for its whole context every turn, with production calls running 92% carried context by input token. The default model's session is also **immutable**, so mid-call notes and supervisor guidance cannot reach it: the worker records the attempt as undelivered rather than pretending it landed. Choosing the 2.5 model buys mid-call steering back at roughly a second per reply.

> **Careful** The floor on this path is the model's first token plus its end-of-turn detection: **3.3 s median reply on the best measured call**. Settings move that; nothing removes it.

## Questions

### Do I have to move to the realtime engine to use Connect on the phone?

No. The turn-based carrier path is the default and needs no trunk, no worker and no fleet to run. The realtime engine is what you choose when being interruptible matters more than simplicity.

### Can a colleague steer a live call on this engine?

Guidance is delivered through the supervisor panel and never reaches the caller — it goes into the system prompt and nowhere else. Whether it reaches the model at all depends on the model: the default 3.1 session is immutable and the attempt is recorded as `guidance_undeliverable`, while the 2.5 model accepts it.

### What happens if the model refuses to start a session?

It is retried once on the same model and once on the mutable fallback before the call is given up, and `model_session_restarted` goes on the record. A model id that is a Vertex path or otherwise malformed is refused up front as `bad_model`, before a row exists.

## Related

- [The carrier turn-based engine](https://connectbyjbrh.com/docs/phone/turn-based-engine/)
- [The voice worker](https://connectbyjbrh.com/docs/phone/voice-worker/)
- [Capacity and admission control](https://connectbyjbrh.com/docs/phone/worker-capacity/)
- [What a call costs](https://connectbyjbrh.com/docs/phone/voice-cost/)
- [What a setting cannot change](https://connectbyjbrh.com/docs/phone/model-limits/)
- [Designing telephony that does not name its provider](https://connectbyjbrh.com/research/provider-independent-telephony/)

## What this page is based on

- AGENTS.md §9a — the realtime engine and the worker boundary (via docs-source/sources/PHONE.md §3, §7)
- PROJECT-STATE.md — first-token and instruction-size measurements (via docs-source/sources/PHONE.md §7)
- `backend/app/voice_engine.py` — `brief`, `incoming`, `ring`
- `backend/voice_worker/` — the `connect-voice` worker