# OpenAI-compatible STT endpoint

The `openai` STT backend keeps VAD, turns, sessions, conversation state, and
response handling inside speech-to-speech while delegating recognition to an
external server:

```text
VAD audio -> POST /v1/audio/transcriptions
```

Each request uploads an in-memory mono PCM16 WAV at 16 kHz and accepts either
JSON with a string `text` field or a plain-text response. With live
transcription enabled, progressive updates upload the accumulated utterance
again, increasing request volume and provider usage.

## vLLM with Qwen3-ASR

Run a supported ASR model behind vLLM. Install vLLM in its own environment on
the machine serving STT; it does not need to be inside this repository.

```bash
pip install "vllm[audio]"
vllm serve Qwen/Qwen3-ASR-1.7B --port 8000
```

Check the server, then select the remote backend. The LLM and TTS flags can
point at any other supported backends.

```bash
curl http://localhost:8000/v1/models

speech-to-speech local \
  --stt openai \
  --openai_stt_base_url http://localhost:8000/v1 \
  --openai_stt_model Qwen/Qwen3-ASR-1.7B
```

### Bounded long turns (opt-in)

Add `--openai_stt_max_window_s 120` to limit every transcription upload to
two minutes of audio. The same setting is available in a configured STT block.
The value must be finite and at least one second, and should be below your
server's duration and upload limits. Leaving it unset preserves whole-turn uploads.

The handler uses VAD's
[progressive audio snapshots](../src/speech_to_speech/VAD/README.md#progressive-audio-snapshots)
to transcribe bounded windows while speech continues. Completed text is reused
across reopened speech; at turn end, remaining audio is transcribed and the full
transcript assembled. Internal cuts do not trigger an assistant response.

Cuts use a quiet point up to five seconds before the limit, without overlapping
or discarding audio. They stay bounded even without silence, but can split words
and affect recognition.

Processing also works with live captions disabled; `--enable_live_transcription`
controls partial transcript output. VAD still retains turn audio, so this option
does not bound its memory usage.

## Native Nemotron 3.5 Transcribe prompts (opt-in)

For a vLLM server hosting Nemotron 3.5 Transcribe, select the `nemotron`
profile to send audio and recognition instructions through `/v1/chat/completions`.
This is an STT request; the conversation LLM remains separate. The default
`transcription` profile continues to use multipart `/v1/audio/transcriptions`.

```bash
speech-to-speech local \
  --stt openai \
  --openai_stt_base_url http://localhost:8000/v1 \
  --openai_stt_model nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B-NVFP4 \
  --openai_stt_profile nemotron \
  --openai_stt_max_window_s 30 \
  --openai_stt_recognition_context "This audio was captured by a voice assistant running inside a robot. These names may occur frequently. This is context for recognition, not evidence that every utterance addresses the robot." \
  --openai_stt_context_terms "Reachy Mini" "Pollen Robotics" "Hugging Face"
```

Recognition context and vocabulary are optional and have no robot-specific
defaults. Names are listed once, followed by an instruction to use their
spellings only when supported by the audio. This is recognition biasing, not
keyword spotting or a guarantee against hallucinations. It does not identify
the main speaker or whether speech addresses the assistant.

The profile asks for verbatim speech with Nemotron's non-speech and style
annotations. Only its known event, style, and speaker tokens are removed before
text reaches captions, conversation history, or the LLM. An event-only final
still closes the transcription item with empty text and creates no LLM work.
Mixed output such as `<ev:gesture_noise> Uh-huh.` retains `Uh-huh.`. Real
interjections are never blacklisted, so a word hallucination without an event
token can still pass through. Keep VAD enabled.

The session's `audio.input.transcription.language` selects a language hint,
for example `fr` adds `Language: French. Do not translate.`. Explicit `auto`
removes the hint, including a configured `--openai_stt_language`. Language
names use the shared ISO registry rather than a TTS-specific language list;
the served model must still support the selected language. Auto results carry
no detected-language claim because audio chat does not report one.

Requests disable thinking, retain annotation tokens, use temperature zero,
and request one non-streaming result. Invalid, reasoning-bearing, or truncated
responses fail the transcription instead of creating LLM work. The default
`--openai_stt_max_output_tokens 384` may need increasing for dense speech or
longer audio windows. Model, prompt, language and token-limit failures are
detected by the setup silence request before accepting sessions. Existing
HTTP cancellation and bounded-window behavior also apply to this profile.

A routing proxy must support the audio-chat route with STT accounting; merely
forwarding a multipart `prompt` field to a fixed server transcription template
does not activate these instructions.

## OpenAI-hosted transcription

The same backend can call OpenAI's hosted
[Transcription API](https://developers.openai.com/api/docs/guides/speech-to-text).
Export an API key and select a transcription model:

```bash
export OPENAI_API_KEY=...

speech-to-speech local \
  --stt openai \
  --openai_stt_base_url https://api.openai.com/v1 \
  --openai_stt_model gpt-transcribe \
  --openai_stt_response_format json
```

The LLM and TTS backends remain independently configurable; using OpenAI for
STT does not require using it for the rest of the pipeline.

## Authentication and compatibility

Set `--openai_stt_api_key` when the endpoint requires bearer authentication.
When the base URL is `https://api.openai.com/v1` and this flag is omitted, the
handler uses `OPENAI_API_KEY` if present. Other endpoints never receive that
environment credential implicitly.

The request and response shapes follow OpenAI's Audio API. In particular,
`gpt-transcribe` sends language hints with the official plural `languages[]`
field and reads the first detected language code from the plural `languages`
response. Older models and compatible servers continue to use the singular
`language` field.

The client accepts JSON and text responses. Use
`--openai_stt_response_format text` for a plain-text server. Transport and HTTP
errors are sanitized before they are surfaced to realtime clients, and failed
final requests do not create LLM work.

During setup, the handler transcribes one second of synthetic silence through
the configured endpoint. Endpoint, authentication, model, or response-format
failures therefore prevent the realtime server from accepting sessions.

## Turn and session lifecycle

Each pipeline delivers STT results asynchronously so HTTP work does not block
session teardown. Final requests for distinct turns retain their order within
that pipeline. They run independently of progressive work. Each pipeline retains
at most eight pending finals in addition to its active final request. Once this
queue is full, additional finals receive a sanitized `TranscriptionFailure`
without uploading audio or retrying. Obsolete pending requests are removed before
checking the limit, and accepted finals keep their order. This bounds the number
of utterances retained behind a stalled request; it is not an estimate of server
capacity.

Progressive requests are best-effort: one is active per pipeline, and only the
latest waiting cumulative window is retained. A final request cancels matching
active progressive work and discards its pending window. Newer turn revisions
invalidate older queued work and cancel older active requests. Relevance is
checked before HTTP dispatch and while a request is running.

Session end cancels active work and clears pending work. Results and failures
carry a session generation that is checked atomically with queue publication,
preventing old completions from appearing after teardown or in a reused session.
Shutdown cancels the remaining requests and joins the pipeline's request workers.
If a request worker cannot start, its pending work is cleared, final requests
receive a sanitized failure, and shutdown still reaches the pipeline boundary.

Cancellation remains active through connection establishment, upload, and
response reads, even if a connection handoff consumes an initial cancellation
signal. Transport cleanup completes before the request worker is reused, so an
old blocked upload does not hold up a new session until its HTTP timeout.
Server-side inference cancellation is best-effort: closing the client connection
does not guarantee that the server stops GPU work. Stale-result filtering remains
required even when a request has been cancelled.

Client-side queue bounds apply separately to each pipeline. Pipelines do not
share an admission queue or endpoint concurrency budget. STT server capacity,
fleet routing, and provider quotas belong to the inference service or its shared
proxy.

## Stateful streaming STT

Use `--stt openai-realtime` or `--stt vllm-realtime` to send incremental PCM
over a persistent WebSocket. Local VAD controls which audio is sent and when
an utterance is committed. Streaming preserves the configured pre-speech
padding, trailing silence, and short-fragment merge gaps; idle microphone
audio outside those windows is not uploaded. Smart Turn and speculative
reopening still control when the assistant responds.

Both streaming backends forward provider partial transcripts as they arrive,
regardless of `--enable_live_transcription`. That flag enables repeated
whole-utterance requests for non-streaming STT backends; it is not needed for
native streaming.

### OpenAI Realtime transcription

```bash
export OPENAI_API_KEY=...

speech-to-speech local \
  --stt openai-realtime \
  --openai_realtime_stt_model gpt-live-transcribe
```

The hosted endpoint uses a transcription session with `intent=transcription`;
the model is configured in the session, not in the WebSocket URL. Hosted
OpenAI requires 24 kHz PCM, so the handler resamples the pipeline's 16 kHz
audio. Leave `--openai_realtime_stt_audio_sample_rate` at its default of
`24000`; other values fail during setup. See the
[OpenAI Realtime transcription guide](https://developers.openai.com/api/docs/guides/realtime-transcription).

`--openai_realtime_stt_api_key` accepts an explicit bearer token. The default
hosted endpoint uses `OPENAI_API_KEY` when that option is omitted; custom
endpoints do not receive the environment key implicitly.

### vLLM Realtime transcription (experimental)

The client uses vLLM's separate Realtime transcription protocol with 16 kHz
PCM. `--vllm_realtime_stt_model` must match a Realtime-capable model identifier
served by your endpoint; check `/v1/models`, including any custom served name.
The default `Qwen/Qwen3-ASR-1.7B` only works when that identifier is served.

For a server serving `mistralai/Voxtral-Mini-4B-Realtime-2602`:

```bash
curl http://localhost:8000/v1/models

speech-to-speech local \
  --stt vllm-realtime \
  --vllm_realtime_stt_base_url ws://localhost:8000/v1 \
  --vllm_realtime_stt_model mistralai/Voxtral-Mini-4B-Realtime-2602
```

Set `--vllm_realtime_stt_api_key` if the server requires authentication. See
[vLLM's Realtime API documentation](https://docs.vllm.ai/en/stable/serving/online_serving/speech_to_text/#realtime-api)
for supported models and server configuration.