realtime-gateway

A local /v1/realtime endpoint where a small audio model runs the conversation and a separately configured model does the reasoning, lookups and tool work.

client ── /v1/realtime?model=local-live ──> gateway (this repo)
                                              │  hidden: private `delegate` tool
                                              ▼
                              LFM3-Audio  /v1/realtime?model=lfm-audio   (voice)
                                              │  function_call delegate(request)
                                              ▼
                              brain       /v1/chat/completions?model=lfm  (reasoning + tools)
                                              │  {"status": "...", "say": "..."}
                                              ▼
                              function_call_output ──> voice speaks `say`

The voice's tool list, its delegate calls, the brain's identity and the internal response ids stay private. Tools the client declares are the client's own, so a brain call to one surfaces as a realtime function_call item and the client answers it with function_call_output.

What it does:

• One response per turn. A hidden delegate cascade makes the audio model and the brain look like one speaker.
• Server VAD always on. speech_started / speech_stopped, automatic commit and response.create, barge-in with conversation.item.truncated. The client streams mic audio and nothing else.
• Client tools. session.update {tools} -> function_call out -> function_call_output in -> the parked brain call resumes.
• Live ASR. A transcription session beside the voice gives the brain text for spoken turns.
• Rejects what it does not implement instead of pretending.

Quick start

bun install
bun run setup:vad                       # fetches TEN VAD (not committed)
LLAMA_API_KEY=... bun src/server.ts     # http://127.0.0.1:8002

Needs a realtime audio model (lfm-audio) and a chat model (lfm), on one llama.cpp router or separate servers via VOICE_URL / BRAIN_URL / ASR_URL. Full environment reference: docs/deployment.md.

Probes

export LLAMA_API_KEY=...                 # or GATEWAY_API_KEY
bun run probe                            # text turn
bun run probe --wav request.wav          # spoken turn (PCM16 WAV, any rate)
bun run probe:tools                      # client tool round trip
bun run probe:tools --wav request.wav
bun run probe:barge --speak talk.wav     # talk over the answer
bun run bench --turns 5 [--tools] [--wav q.wav] [--parallel N]

Probes write runs/probe-<timestamp>/ (events.jsonl, reply.wav, result.json) and fail if the public stream contains an internal marker. probe:barge needs real speech; TEN VAD does not classify tones as speech.

Docs

• Protocol — events, server VAD, barge-in, client tools, turn assembly, limits
• Context — what the voice and the brain each remember
• Benchmarks — latency, concurrency, numbers from this host
• Findings — what breaks, what is still missing
• Deployment — models, URLs, env, slots, logs
• TEN VAD in Bun — the vendored detector and its gotchas

Status: 71/71 protocol tests, tsc --noEmit clean. This is a deliberately small subset of OpenAI's Realtime API, not GPT-Live. No echo cancellation in the gateway (use client-side AEC or headphones), and no async delegation yet.

License

MIT for this repository. TEN VAD is not vendored: bun run setup:vad downloads it under its own license (Apache-2.0 plus conditions). See THIRD_PARTY_NOTICES.md.