---
title: HF Realtime Voice
emoji: 🎙️
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
short_description: Voice chat over WebSocket or WebRTC against HF speech-to-speech
hf_oauth: true
hf_oauth_expiration_minutes: 10080
---

# Realtime Voice Demo

Browser voice-chat UI for the
[huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech)
backend, which runs locally by default. The UI speaks the OpenAI Realtime **GA** protocol over **WebSocket**
(default) or **WebRTC** (Settings → Transport, env-pinned deploys only — see
[WebRTC transport](#webrtc-transport)).

Both choices run one `RealtimeSession` adapter over the pinned official
`@openai/agents` package's stock transport classes. The adapter keeps demo-only
queue, audio, visualization, device, camera, and metering behavior out of the
protocol implementation.

## Quick start (local)

You need a repository checkout, [uv](https://docs.astral.sh/uv/getting-started/installation/), and Node.js/npm. See the [main README](../README.md#quickstart) for platform notes.

1. **Start the speech server** from the repository root:

   ```bash
   uv sync --python 3.11
   uv run speech-to-speech serve
   ```

   The first start downloads the models. Wait until the server listens at
   `ws://127.0.0.1:8765/v1/realtime`.

2. **Install the pinned browser SDK and start this app**, pointing it at the
   backend with `SPEECH_TO_SPEECH_URL`:

   ```bash
   npm ci --prefix demo
   uv pip install -r demo/requirements.txt
   export SPEECH_TO_SPEECH_URL=ws://localhost:8765/v1/realtime
   uv run uvicorn --app-dir demo server:app --reload --port 7860
   ```

   Or with Docker:

   ```bash
   docker build -t s2s-demo demo/
   docker run -p 7860:7860 -e SPEECH_TO_SPEECH_URL=ws://host.docker.internal:8765/v1/realtime s2s-demo
   ```

   > **Docker + host backend: WebSocket and WebRTC need different hostnames.**
   > The two transports dial the backend from different network namespaces:
   >
   > - **WebRTC** is dialed **server-side** — the browser POSTs its SDP offer to
   >   the demo's `/api/calls` proxy, which forwards it from *inside the
   >   container*. There `host.docker.internal` resolves to your host, so the
   >   command above works.
   > - **WebSocket** is dialed **client-side** — the demo hands the URL straight
   >   to the browser, which opens the socket itself. The browser runs on your
   >   *host*, where `host.docker.internal` is not a real DNS name, so the
   >   connection never reaches the backend and the server logs nothing.
   >
   > A single Docker `SPEECH_TO_SPEECH_URL` can therefore only make one transport
   > work at a time (`localhost:8765` for WebSocket, `host.docker.internal:8765`
   > for WebRTC). To exercise **both** without swapping the env, run the demo
   > **without Docker** (the `uvicorn` command above) so host and container
   > namespaces collapse — then `ws://localhost:8765/v1/realtime` works for both.

3. Open <http://localhost:7860/>, click the orb, allow the mic, talk.

To change models, configure the speech server (see the
[LLM guide](../src/speech_to_speech/LLM/README.md)). Optional [web search](#tools) and
[startup greetings](#startup-greeting) are covered below.

> Browsers require **HTTPS or `localhost`** for `getUserMedia()` (mic + camera).
> `127.0.0.1` and `localhost` both work; plain `http://192.168.x.y` does NOT.

Smoke-test the backend from the shell:

```bash
websocat ws://localhost:8765/v1/realtime
# -> you should get a session.created event back immediately
```

## Response timings

Open Conversation and expand **Server timings** beneath a response to inspect
E2E (estimated speech end to first generated audio), VAD end decision, Smart Turn
decision, transcription, response generation, voice synthesis to first audio,
and hold time before response. E2E is visible in the collapsed summary.
MLX lock wait is available only in macOS terminal logs.
Tool-only responses receive their own timing entry; follow-ups do not repeat STT.

These are server measurements, excluding browser buffering and playback. Stages
can overlap and must not be summed. Missing stages display **Unavailable**. Older
servers, absent metadata, and unsupported or malformed records leave the transcript
unchanged. Server timings become available when the response finishes, including
interrupted, failed, and incomplete responses.

Validation: `npm run test:agents:adapter` checks parsing and adapter behavior;
`npm run test:ui` checks the history display at desktop and phone widths (requires
`npx playwright install chromium`).

## How it works

1. The adapter creates an official Agents SDK `RealtimeSession` with the stock
   WebSocket transport on the configured `/v1/realtime` URL.
2. The SDK performs `session.update` using the OpenAI Realtime **GA** schema.
3. The SDK streams mic audio as PCM16 24 kHz mono base64 chunks
   (`input_audio_buffer.append`, one frame every ~40 ms).
4. The browser keeps a bounded, in-memory copy of those sent frames and uses
   the server VAD boundaries to add replayable user recordings to conversation
   history. No recording is uploaded or persisted separately.
5. Server pushes `response.output_audio.delta` (PCM16 24 kHz mono base64)
   and transcript deltas.

The backend exposes one concurrent session per pipeline unit
(`--num_pipelines` to serve more).

## WebRTC transport

With `SPEECH_TO_SPEECH_URL` set, **Settings → Transport** offers WebRTC as an
alternative to the WebSocket. Same conversation, different plumbing:

1. The SDK's stock WebRTC transport adds the browser mic track and its data
   channel to an `RTCPeerConnection` and POSTs the SDP offer to the same-origin
   `/api/calls` proxy, which forwards it to the backend's
   `POST /v1/realtime/calls` (the OpenAI GA handshake). The proxy exists
   because the s2s server has no CORS middleware — and it forwards **only**
   to the env-pinned URL, never to a client-supplied one, so it can't be used
   as an open proxy. That's why the toggle is locked to WebSocket when the
   URL isn't pinned (user-typed URLs, LB mode).
2. Only the handshake goes through the proxy: the negotiated audio (Opus RTP
   both ways) and the data channel flow directly browser ↔ backend.
3. JSON events on the data channel are the same GA protocol as the WebSocket,
   minus the audio: mic audio rides the media track (never
   `input_audio_buffer.append`, which the backend rejects over WebRTC), and
   the assistant's voice arrives as a remote audio track (never
   `response.output_audio.delta`). Barge-in flushing is server-side.

Backend requirement: the `webrtc` extra
(`pip install "speech-to-speech[webrtc]"`), otherwise `/v1/realtime/calls`
answers 501 and the handshake fails with a clear message.

Caveats vs. WebSocket:

- **User recording replay**: conversation-history recordings currently use the
  exact PCM frames sent through `input_audio_buffer.append`, so they are
  available only on the WebSocket transport.
- **NAT**: host ICE candidates only by default — fine when browser and backend
  are on the same machine/LAN. Across the internet, set `RTC_ICE_SERVERS` on
  *this* app (a JSON list of `RTCIceServer` dicts, or comma-separated
  STUN/TURN URLs; served to the browser via `/api/config`) and
  `SPEECH_TO_SPEECH_ICE_SERVERS` on the backend. There is no TURN relay
  fallback, so symmetric-NAT setups may still not connect.
- **Noise gate**: implemented in the WebSocket capture worklet, so it's
  hidden on WebRTC — the raw mic track (with the browser's own
  `noiseSuppression`) is sent instead.
- **Camera snapshots** are re-encoded to fit one data-channel message
  (~60 KB), so the model may see a smaller frame than over WebSocket.
- **Load-balancer mode is WebSocket-only** for now.

## Faster interruptions

**Settings → Faster interruptions** is on by default for WebSocket and WebRTC.
A browser-local Silero v6 VAD watches the existing echo-cancelled microphone
stream in 32 ms frames. A speech candidate immediately lowers assistant playback
to 25% volume, including audio already queued in the browser. Two quiet frames
(64 ms) restore normal volume if the candidate is rejected. This follows
[Reachy's local barge-in approach](https://github.com/pollen-robotics/reachy_mini_conversation_app#configuration).

Server VAD still confirms interruptions, cancels generation, and controls turn
boundaries. Local detection does not commit input audio or request responses,
and the microphone continues streaming normally. Muting the mic also disables
local ducking. Changes to the setting apply on the next conversation or Restart.

The pinned `@ricky0123/vad-web` model and `onnxruntime-web` WASM assets are served
by this demo (installed with `npm ci`, included in Docker). Inference runs on the
device without an external model CDN. Loading happens in the background; a load
failure falls back to server VAD. Missing inference frames restore volume after
500 ms. Speaker echo and background speech can still cause unwanted ducking;
disable the setting if needed.

Validation: `npm run test:agents:adapter` covers detection, lifecycle, and playback
regressions. `npm run test:ui` runs real Silero inference on a speech recording in
Chromium as well as the history UI checks (`npx playwright install chromium`).

## Connecting to a backend

Three modes, picked by env (`/api/config` tells the client which one is active):

- **`SPEECH_TO_SPEECH_URL` env** — the mode you want for local use, and the
  highest priority. The browser connects **directly** to this realtime
  WebSocket URL; it's shown read-only in Settings. Setting it disables the
  load-balancer logic entirely (no `/api/session` proxy, no queue, no
  metering, no sign-in). Unlike the LB address it is not a secret. Accepts a
  full `ws(s)://host/v1/realtime` URL or a bare host like `localhost:8765`
  (the app adds `/v1/realtime`).
- **Neither env set** — **Settings → Speech-to-speech server URL**: paste a
  full connect URL or a bare host, and the browser connects to it directly.
- **`LOAD_BALANCER_URL` env** — multi-compute deployments only: the browser
  POSTs the same-origin `/api/session` proxy, the server forwards to the LB,
  and the browser dials the per-session compute URL the LB hands back. The LB
  address never reaches the browser; the Settings URL field is hidden. On
  OAuth-enabled Spaces, the proxy forwards the signed-in user's HF access token
  to the LB through `X-Reachy-Mini-Authorization` so the backend can attribute
  usage. The token stays server-side; anonymous requests include no credential.

| `SPEECH_TO_SPEECH_URL` | `LOAD_BALANCER_URL` | `SPACE_ID` | Connection | URL field | Transport | Metering |
|:---:|:---:|:---:|---|---|---|---|
| ✅ | any | any | direct → pinned URL | visible, locked | WS or WebRTC | off |
| – | – | any | direct → user URL | editable | WS only | off |
| – | ✅ | ✅ | LB proxy | hidden | WS only | **on** |
| – | ✅ | – | LB proxy | hidden | WS only | off |

**Settings → Restart** reconnects with the current voice, instructions and URL.

## Startup greeting

By default, each new connection creates one hidden user item asking the model
for a brief greeting, then requests a response. Besides opening the conversation
naturally, this warms the same prompt prefix used by the first spoken turn.

Set `STARTUP_GREETING` to customize the hidden prompt, or set it to an empty
value to disable automatic generation. The adapter sends it once through the
new `RealtimeSession` after connection.

## Tools

The assistant can call two tools mid-conversation (toggle them from the **Tools**
button, top-right):

- **Web search** — Google results via Serper.dev, proxied server-side so the key
  never reaches the browser. Set `SERPER_API_KEY` as an env var / Space secret.
  Without it, the tool is disabled unless the user pastes their own key in the
  Tools panel. On a Space the server key is only spent for a signed-in HF user
  with a live conversation (every tier), capped at `LIMIT_SEARCH_PER_DAY`
  searches per user per day; a Space without HF sign-in never uses it. Locally
  the key is unrestricted.
- **Camera** — while enabled, a live self-view shows bottom-left; when the model
  calls the tool, the current frame is sent to the vision-language model so it can
  see what you're showing it. This needs a vision-language model on the server
  (`--llm_is_vlm`; the default model is text-only); see the
  [LLM guide](../src/speech_to_speech/LLM/README.md#local-generation-options).

## Usage limits (deployed Space only)

Conversation time is metered per UTC day by sign-in tier (see `limiter.py` /
`auth.py`), but **only on the deployed Space** — metering turns on only when BOTH
`LOAD_BALANCER_URL` and `SPACE_ID` (injected automatically by the HF Space
runtime) are present. Running locally — even with `LOAD_BALANCER_URL` exported —
leaves the app unmetered. Tunable via env:

| Env | Default | What |
|-----|---------|------|
| `LIMIT_ANON_SEC` | `300` | Daily seconds for anonymous visitors (5 min) |
| `LIMIT_FREE_SEC` | `600` | Daily seconds for signed-in non-PRO users (10 min) |
| `LIMIT_SEARCH_PER_DAY` | `100` | Daily web searches on the server's Serper key per signed-in user (every tier) |
| `LB_HF_TOKEN` | _(falls back to user OAuth)_ | Optional Space secret sent in standard `Authorization` to authenticate requests at the HF Inference Endpoint ingress; per-user attribution continues through `X-Reachy-Mini-Authorization` |
| `UNLIMITED_ORGS` | _(adds to defaults)_ | Extra HF org names whose members get **unlimited** usage, like PRO |
| `USAGE_HASH_SECRET` | _(random)_ | HMAC secret for hashing identity keys + signing the anon cookie |

PRO members are always unlimited. Members of `cerebras`, `HuggingFaceM4`,
`smolagents`, and `pollen-robotics` are unlimited out of the box (shown as
"Team", not "PRO"); set `UNLIMITED_ORGS=my-team` to add more. Matched
case-insensitively against the user's organisations from HF OAuth.

## Settings (stored in `localStorage`)

| Key | What |
|-----|------|
| Speech-to-speech server URL | Direct realtime WebSocket URL (hidden/locked when pinned by env) |
| Transport | WebSocket (default) or WebRTC; selectable only with an env-pinned URL |
| Microphone | Input device for capture. Applies on the next conversation / Restart. |
| Speakers | Output device for assistant audio. Chrome/Edge can switch live via `AudioContext.setSinkId`; other browsers keep the system default. |
| Voice | Qwen3-TTS speaker name (Aiden, Ryan, Dylan, Eric, Ono_Anna, Serena, Sohee, Uncle_Fu, Vivian) |
| Show reasoning | Show or hide the reasoning and summary sections in Conversation. Off by default; when enabled, each section starts collapsed. Applies immediately to existing and future replies and is saved in this browser. This changes display only; the reasoning effort is selected separately. |
| Reasoning (button below the orb) | Server default, Off, Minimal, Low, Medium, High, or Extra high. Saved in this browser and applied to the next response without reconnecting, over either transport. The line below the button waits for `session.updated` and shows a green “Server confirmed” only when the server echoes the selected level. Missing or different levels stay unconfirmed. This verifies the server's session setting; model support determines how it is used. Higher levels may increase response time. Server default clears the override and restores the backend's startup setting. |
| Instructions | System prompt sent in `session.update` once the connection opens |

LocalStorage keys are namespaced `s2s.ws.*` (plus `s2s.transport` for the
transport pick, and `s2s.audio.inputId` / `s2s.audio.outputId` for devices).

Enable **Settings → Show reasoning**, then open **Conversation** (the chat icon)
to read the provider's reasoning under
each reply. **Reasoning** and **Reasoning summary** sections are collapsed by
default; expand one to read it. On every setup, reasoning is collected in the
LLM handler and delivered with the first answer text or tool call, rather than
streamed live. A response with neither delivers its reasoning at completion.
Reasoning stays separate from the answer and is never sent to speech synthesis.
A section appears only when the model/provider supplies readable reasoning or
a summary; encrypted reasoning is not displayed. This needs the updated
speech-to-speech backend as well as the demo frontend.

Turn **Settings → Show reasoning** off to hide these sections, or back on to
restore them. The preference is saved and applies without restarting a call.

Reasoning alone does not commit a turn or close the user's chance to resume a
paused utterance. Reasoning from a cancelled or superseded generation is not
sent.

## Files

| File | Role |
|------|------|
| `index.html` | Single page, orb + settings modal (identical UI to the WebRTC app) |
| `main.js` | State machine, settings, tools, camera, noise-gate UI wiring |
| `ui/chat.js` | `ChatView`: history panel, ephemeral bubbles, transcript/tool streaming, user recording replay |
| `ui/account.js` | `Account`: HF login chip + popover, daily-limit modal |
| `ui/dom.js` | Shared helpers: `$`, `escHtml`, `truncateError`, `DEBUG` |
| `auth.py` | HF OAuth + per-request identity (tier, hashed keys) |
| `limiter.py` | SQLite per-day talk-time budget (chunked server-clock reservation) |
| `s2s-realtime-client.js` | Narrow demo adapter around one Agents SDK `RealtimeSession` and SDK transports with browser-owned WebSocket truncation |
| `package.json` / `package-lock.json` | Exact official Agents SDK and browser-test dependency pins |
| `ws/codec.js` | URL normalization + transcript extraction (pure) |
| `ws/local-vad.js` | Browser Silero detection, reversible playback ducking, and lifecycle |
| `ws/user-audio-recorder.js` | Bounded sent-PCM buffer + VAD slicing + browser-playable WAV wrapping |
| `ws/orb-visualizer.js` | `OrbVisualiser`: FFT bands -> orb CSS custom properties |
| `worklets/mic-capture.js` | AudioWorklet: 48 kHz Float32 -> 24 kHz Int16 PCM, posts ~40 ms chunks |
| `worklets/audio-playback.js` | AudioWorklet: 24 kHz Float32 ring buffer -> 48 kHz, linear interp, fade in/out |
| `style.css` | Orb animations, layout, dark theme (verbatim from the WebRTC app) |

## Audio pipeline notes

- **WebSocket startup buffer**: Settings → Playback startup buffer (ms) controls
  how much assistant audio is accumulated before each response starts playing.
  The default is 0 (immediate playback). The setting is saved in this browser
  and applies on the next conversation; it does not affect WebRTC or the Python
  client's `--playback-buffer-ms` setting. Missing, invalid, or negative values
  fall back to the default.
  Buffering counts PCM samples, not elapsed time. After the threshold is reached,
  later chunks stream immediately without rebuffering. Completed short responses
  and valid incomplete responses release their remaining audio; interruption,
  cancellation, failure, and disconnect discard pending audio. Each new response
  gets a fresh startup gate without cutting off already released audio. A failed
  or cancelled response discards only its private startup buffer; already released
  audio drains unless an interruption or disconnect clears the shared queue.
  In [issue #557](https://github.com/huggingface/speech-to-speech/issues/557),
  1200 ms resolved glitches in the reporter's local TTS setup. This is a tuning
  example, not a universal optimum: larger values add startup latency, and no
  finite reserve can prevent all underruns from sustained slower-than-realtime
  generation.
- **Input**: `getUserMedia({ echoCancellation, noiseSuppression, autoGainControl })`
  feeds the `mic-capture` worklet at the `AudioContext` rate. The worklet
  resamples to 24 kHz (boxcar lowpass + decimation on the 48 -> 24 fast
  path, linear interpolation fallback for odd rates) and packs Int16 LE.
- **Browser cache safety**: the entry module, realtime client, and both audio
  worklet URLs share the `reasoning-language-20261110` cache key. The client also waits for
  the capture worklet to report the same version and a 24 kHz output rate
  before opening a session. When the browser-audio contract or sample rate
  changes, bump the key in `index.html`, `main.js`, and
  `AUDIO_WORKLET_VERSION` in `s2s-realtime-client.js` together.
- **User replay**: the WebSocket client retains only a bounded copy of PCM it
  actually sends. `speech_started` / `speech_stopped` timestamps select each
  utterance, which is wrapped as an in-memory WAV and attached to the user row.
  Starting playback temporarily mutes outgoing mic audio to prevent feedback.
- **Output**: `response.output_audio.delta` decodes to Int16 -> Float32
  and is posted to the `audio-playback` worklet. The worklet maintains a
  per-context ring buffer, linearly interpolates 24 -> 48, and applies
  short 32-frame fades on entry/exit to suppress clicks.
- **Barge-in**: server VAD and explicit SDK interruptions clear the WebSocket
  playback queue. The worklet acknowledges the clear with rendered PCM counts
  per item/content part, excluding startup delay and underrun silence. The client
  sends `conversation.item.truncate` for unheard audio using those counts (zero
  before playback starts), retaining identities even after audio/response done.
  This overrides the SDK's receipt-time interruption clock. Server VAD cancels
  the in-flight response; explicit interruptions use the SDK cancellation hook.
  The server cuts the item's model history to the transcript heard in that
  time; see [Interrupted replies](../src/speech_to_speech/api/openai_realtime/README.md#interrupted-replies).
  After each truncate the client sends `conversation.item.retrieve`. The
  reply's history row then strikes through the words never said, marked
  Interrupted, and its bubble, while still on screen, draws the strike across
  them (statically with reduced motion). Over WebRTC, where the server measures playback, the client
  retrieves the items of the last two replies after each barge-in. Only an
  item the server cut (status `incomplete`, or removed) changes its row.
  See the [Realtime interruption guidance](https://developers.openai.com/api/docs/guides/realtime-conversations#handling-interruptions).

## Credits

- Backend: [huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- UI verbatim from `amir-tfrere/minimal-conversation-app-s2s-backend` (Pollen Robotics × Hugging Face)

Choose **Language** on the conversation screen or in Settings to set transcription and response language. The choice persists across reloads and can change during a conversation; **Auto** restores automatic language selection. Language support depends on the configured speech backends.

The demo checks backend language capabilities before connecting. When Auto is unsupported or discovery is unavailable, it preserves the backend default and disables switching back to Auto during that conversation. End the conversation to select backend defaults for the next one. Language changes while queued apply when the conversation starts.