# Shared Protocols: Model-Agnostic Interfaces

## Overview

The `AudioCommon` module defines shared protocols that provide model-agnostic interfaces for speech processing. These allow generic code to work with any conforming model without knowing its concrete type.

```
┌─────────────────────────────────────────────────────────┐
│                    AudioCommon                          │
│                                                         │
│  AudioChunk / CapturedAudioChunk                        │
│                      SpeechGenerationModel (TTS)        │
│  AlignedWord         SpeechRecognitionModel (STT)       │
│                      StreamingRecognitionModel          │
│                      StreamingRecognitionSession        │
│  SpeechSegment       ForcedAlignmentModel                │
│  TranscriptionResult SpeechToSpeechModel                 │
│                      VoiceActivityDetectionModel (VAD)   │
│                      StreamingVADProvider (pipeline)      │
│                      TurnCompletionProvider (pipeline)    │
│                      SpeakerEmbeddingModel               │
│                      SpeakerDiarizationModel             │
│                      SpeakerExtractionCapable            │
│                      SpeechEnhancementModel              │
└─────────────────────────────────────────────────────────┘
        ▲                    ▲                    ▲
        │                    │                    │
   ┌────┴────┐        ┌─────┴─────┐       ┌─────┴─────┐       ┌─────┴─────┐
   │Qwen3TTS │        │  Qwen3ASR │       │PersonaPlex │       │ SpeechVAD │
   │CosyVoice│        │ParakeetASR│       └───────────┘       └───────────┘
   │VoxCPM2  │        │ForcedAlign│
   │Kokoro   │
   │IndexTTS2│
   │F5TTS    │
   │HiggsTTS │
   └─────────┘        └───────────┘
```

## Protocols

### SpeechGenerationModel (TTS)

Text-to-speech models that generate audio from text.

```swift
public protocol SpeechGenerationModel: AnyObject {
    var sampleRate: Int { get }
    func generate(text: String, language: String?) async throws -> [Float]
    func generateStream(text: String, language: String?) -> AsyncThrowingStream<AudioChunk, Error>
}
```

**Conforming types:** `Qwen3TTSModel`, `CosyVoiceTTSModel`, `VoxCPM2TTSModel`, `KokoroTTSModel`, `IndexTTS2TTSModel`, `F5TTSModel`, `HiggsTTSModel`

`IndexTTS2TTSModel` implements bundle loading, manifest validation, metadata access, and `ModelMemoryManageable`. It exposes a reference-audio `generate` overload for the expanded IndexTTS2 bundle and runs native reference conditioning, optional `IndexTTS2EmotionControl` preset/vector blending, `IndexTTS2SynthesisOptions` speaking-rate and internal-pause controls, semantic GPT beam sampling, S2Mel decoding, and BigVGAN vocoding. The protocol-only `generate(text:language:)` entry point throws a reference-required error because IndexTTS2 is a zero-shot voice-cloning model.

`F5TTSModel` implements local bundle loading, config validation, `ModelMemoryManageable`, and a reference-audio `generate` overload for the exported F5-TTS bundle. The runtime prepares Vocos-style reference mels, samples target mels with the native DiT flow model, and decodes 24 kHz waveform audio with Vocos. The protocol-only `generate(text:language:)` entry point throws a reference-required error because F5-TTS requires reference audio plus a reference transcript.

`HiggsTTSModel` implements bundle loading, `ModelMemoryManageable`, native reference encoding (`encodeReference`), and reference-audio `generate` overloads for the Higgs TTS 3 bundle. The runtime builds a `<|tts|>`-protocol prompt, samples delay-patterned 8-codebook frames with a Qwen3 backbone and fused codebook head, and decodes 24 kHz audio with the embedded Higgs codec. The protocol-only `generate(text:language:)` performs reference-free synthesis in a model-chosen voice; inline control tags ride inside the text.

### SpeechRecognitionModel (STT)

Speech-to-text models that transcribe audio.

```swift
public protocol SpeechRecognitionModel: AnyObject {
    var inputSampleRate: Int { get }
    func transcribe(audio: [Float], sampleRate: Int, language: String?) -> String
    func transcribeWithLanguage(audio: [Float], sampleRate: Int, language: String?) -> TranscriptionResult
}
```

The `transcribeWithLanguage` method has a default implementation that delegates to `transcribe()` with no language detection. Models that detect language (e.g. ParakeetASR via `NLLanguageRecognizer`) override it to return `TranscriptionResult` with a detected language — used by the voice pipeline to forward language to TTS.

**Conforming types:** `Qwen3ASRModel`, `ParakeetASRModel`, `ParakeetStreamingASRModel`, `NemotronStreamingASRModel`, `OmnilingualASRModel` (CoreML), `OmnilingualASRMLXModel` (MLX)

`ParakeetStreamingASRModel` additionally exposes streaming APIs — `createSession()` for long-lived streaming with cache state, `transcribeStream(audio:sampleRate:)` for chunked AsyncSequence output, and per-session `pushAudio` / `forceEndOfUtterance` / `finalize` for fine-grained VAD-driven pipelines. The batch `transcribe` entry point runs internal chunking and remains protocol-compatible.

`NemotronStreamingASRModel` exposes the same shape of streaming APIs (`createSession()`, `transcribeStream`, `pushAudio`, `finalize`) minus `forceEndOfUtterance` — Nemotron has no explicit EOU head, so segmentation is driven by the caller (external VAD or punctuation-boundary heuristic) calling `finalize()` directly.

### StreamingRecognitionModel and StreamingRecognitionSession

Incremental recognizers expose one model-agnostic session contract. A session
owns decoder/cache state, consumes ordered `CapturedAudioChunk` values, and
returns a common partial/final result without reprocessing earlier chunks.

```swift
let source = AudioFileLoader.stream(
    url: input,
    options: AudioFileStreamOptions(targetSampleRate: 16_000))
let session = try model.makeStreamingSession(language: "en-US")

for try await chunk in source {
    for update in try session.push(chunk) {
        print(update.text, update.isFinal)
    }
}
for update in try session.finish() {
    print(update.text, update.isFinal)
}
```

`ParakeetStreamingASRModel`, `NemotronStreamingASRModel`, and
`NemotronStreamingASRMLXModel` conform. `StreamingRecognitionUpdate` preserves
the shared text, final/EOU state, segment index, confidence, language, optional
segment bounds, and word timings. Inputs must already use the session's
`inputSampleRate`; capture and file sources can resample before delivery.

### ForcedAlignmentModel

Models that align text to audio at the word level.

```swift
public protocol ForcedAlignmentModel: AnyObject {
    func align(audio: [Float], text: String, sampleRate: Int, language: String?) -> [AlignedWord]
}
```

**Conforming types:** `Qwen3ForcedAligner`

### SpeechToSpeechModel

Speech-to-speech models that generate spoken responses from spoken input.

```swift
public protocol SpeechToSpeechModel: AnyObject {
    var sampleRate: Int { get }
    func respond(userAudio: [Float]) -> [Float]
    func respondStream(userAudio: [Float]) -> AsyncThrowingStream<AudioChunk, Error>
}
```

**Conforming types:** `PersonaPlexModel`

### VoiceActivityDetectionModel (VAD)

Models that detect speech activity regions in audio.

```swift
public protocol VoiceActivityDetectionModel: AnyObject {
    var inputSampleRate: Int { get }
    func detectSpeech(audio: [Float], sampleRate: Int) -> [SpeechSegment]
}
```

**Conforming types:** `PyannoteVADModel`, `SileroVADModel`

### StreamingVADProvider (Pipeline)

Streaming VAD that processes fixed-size audio chunks and returns speech probability. Used by the `SpeechCore` voice pipeline via the C vtable FFI.

```swift
public protocol StreamingVADProvider: AnyObject {
    var inputSampleRate: Int { get }
    var chunkSize: Int { get }
    func processChunk(_ samples: [Float]) -> Float
    func resetState()
}
```

**Conforming types:** `SileroVADModel`

### TurnCompletionProvider (Pipeline)

End-of-turn classifier consulted after the VAD reports a pause. A VAD only hears silence; a turn-completion model listens to the prosody of the whole utterance, so a mid-sentence pause keeps the agent waiting while a finished sentence gets an immediate reply. `StreamingVADProcessor` calls it on every confirmed pause and holds the segment open while the probability stays below `TurnCompletionConfig.threshold`. `VoicePipeline.setTurnCompletion(_:)` attaches it to the speech-core engine with the same semantics (`PipelineConfig.turnCompletionThreshold` / `turnCompletionMaxSilence`); it maps to speech-core's `sc_turn_completion_vtable_t`.

```swift
public protocol TurnCompletionProvider: AnyObject {
    /// Probability in `[0, 1]` that the turn is complete, given the audio of the
    /// turn so far. Implementations look at the most recent seconds (Smart Turn: 8 s).
    func turnCompleteProbability(audio: [Float], sampleRate: Int) throws -> Float
}
```

**Conforming types:** `SmartTurnModel` (Core ML, `SpeechVAD`) — see [Smart Turn model doc](models/smart-turn-v3.md).

### SpeakerEmbeddingModel

Models that extract speaker embeddings from audio.

```swift
public protocol SpeakerEmbeddingModel: AnyObject {
    var inputSampleRate: Int { get }
    var embeddingDimension: Int { get }
    func embed(audio: [Float], sampleRate: Int) -> [Float]
}
```

**Conforming types:** `WeSpeakerModel`

`ReDimNet2SpeakerModel` is the separate persistent identity encoder. Its
`embed` method throws so model or input failures remain explicit; it is used
through its concrete API rather than this legacy non-throwing protocol. Its
explicit `embedShortUtterance` method accepts clean 0.6-to-2-second retrieval
probes, but those lower-evidence vectors must never enroll or update an
identity.

### SpeakerDiarizationModel

Models that assign speaker identities to speech segments.

```swift
public protocol SpeakerDiarizationModel: AnyObject {
    var inputSampleRate: Int { get }
    func diarize(audio: [Float], sampleRate: Int) -> [DiarizedSegment]
}
```

**Conforming types:** `PyannoteDiarizationPipeline` (aliased as `DiarizationPipeline`), `SortformerDiarizer`

### SpeakerExtractionCapable

Extended diarization protocol for engines that support extracting a target speaker's segments using a reference embedding. Not all engines support this — Sortformer is end-to-end and does not produce speaker embeddings.

```swift
public protocol SpeakerExtractionCapable: SpeakerDiarizationModel {
    func extractSpeaker(audio: [Float], sampleRate: Int, targetEmbedding: [Float]) -> [SpeechSegment]
}
```

**Conforming types:** `PyannoteDiarizationPipeline`

### WakeWordProvider (Pipeline)

Streaming keyword / wake-word detector. Shares the chunk-push shape of `StreamingVADProvider` so a voice pipeline can gate activation on either VAD or wake-word triggers (or both). Lives in `SpeechWakeWord` rather than `AudioCommon` because it carries the `KeywordDetection` result type and a session-scoped BPE context.

```swift
public protocol WakeWordProvider: AnyObject {
    var inputSampleRate: Int { get }
    var registeredKeywords: [String] { get }
    func processAudio(_ samples: [Float]) throws -> [KeywordDetection]
    func reset() throws
}
```

**Conforming types:** `WakeWordStreamingAdapter` (wraps `WakeWordDetector` + a single `WakeWordSession`). The underlying `WakeWordDetector` is English-only — see [KWS Zipformer model doc](models/kws-zipformer.md).

### SpeechEnhancementModel

Models that enhance speech by removing noise.

```swift
public protocol SpeechEnhancementModel: AnyObject {
    var inputSampleRate: Int { get }
    func enhance(audio: [Float], sampleRate: Int) throws -> [Float]
}
```

**Conforming types:** `DeepFilterNet3Model`

## Voice Pipeline (SpeechCore)

The `SpeechCore` module provides `VoicePipeline` — a real-time voice agent pipeline powered by [speech-core](https://github.com/soniqo/speech-core) (C++ engine, distributed as xcframework). It connects `SpeechRecognitionModel`, `SpeechGenerationModel`, and `StreamingVADProvider` through a state machine with VAD-driven turn detection, interruption handling, and eager STT.

```swift
import SpeechCore

let pipeline = VoicePipeline(
    stt: parakeetASR,
    tts: qwen3TTS,
    vad: sileroVAD,
    config: .init(mode: .echo),
    onEvent: { event in print(event) }
)
pipeline.start()
pipeline.pushAudio(micSamples)  // feed mic audio continuously
```

### Pipeline Modes

| Mode | Flow | Use case |
|------|------|----------|
| **voicePipeline** | audio → VAD → STT → LLM → TTS → audio | Full voice agent |
| **echo** | audio → VAD → STT → TTS → audio | Testing (speaks back transcription) |
| **transcribeOnly** | audio → VAD → STT → text | Transcription only |

### Configuration

```swift
var config = PipelineConfig()
config.mode = .echo
config.minSilenceDuration = 0.6     // seconds to confirm end of speech
config.eagerSTT = true              // start STT before silence confirms
config.eagerSTTDelay = 0.3          // seconds in silence before eager fires
config.allowInterruptions = true    // user can barge-in during playback
config.minInterruptionDuration = 1.0 // seconds of speech to confirm barge-in
config.maxResponseDuration = 5.0    // cap TTS output (prevents hallucination)
config.postPlaybackGuard = 0.3      // suppress VAD after playback (AEC settle)
config.warmupSTT = true             // warm up Neural Engine at pipeline start
```

### Events

| Event | When |
|-------|------|
| `speechStarted` | VAD confirms user speech |
| `speechEnded` | User utterance finalized |
| `transcriptionCompleted` | STT returns text + language + confidence |
| `responseCreated` | TTS synthesis starting |
| `responseAudioDelta` | TTS audio chunk ready (PCM Float32) |
| `responseInterrupted` | User barged in during playback |
| `responseDone` | TTS synthesis complete |
| `error` | STT/LLM/TTS failure |

## Shared Types

### CapturedAudioChunk

Timestamped mono PCM from live capture. `SystemAudioTap.startTimestamped` and
`AudioIO.startMicrophoneTimestamped` both return this type, using the same Mach
host clock so callers can keep independent source pipelines and merge only
their derived events.

```swift
public struct CapturedAudioChunk: Sendable, Equatable {
    public let samples: [Float]
    public let sampleRate: Int
    public let hostTime: UInt64?
    public let frameIndex: Int64
    public let isFinal: Bool
}
```

`hostTime` identifies the first input frame before capture-side resampling. It
is nil only when the underlying audio callback supplies no valid host time.
For full-duplex capture, construct `AudioIO(enableAEC: true)` to apply Apple's
echo-cancelled microphone input before timestamped chunks reach the caller.
Listen-only clients can additionally pass `enablePlayback: false` to omit the
streaming player from the engine graph.

Finite sources also set `frameIndex` and mark their last chunk with `isFinal`.
`AudioFileLoader.stream` is a pull-driven `AsyncSequence`, so a slow consumer
does not create an unbounded producer queue. Multichannel input is averaged by
default; pass `.first` or `.select([indices])` through
`AudioFileStreamOptions.channelSelection` when channel routing is known.

### AudioChunk

Unified audio chunk type returned by all streaming methods:

```swift
public struct AudioChunk: Sendable {
    public let samples: [Float]    // PCM audio samples
    public let sampleRate: Int     // Hz (e.g. 24000)
    public let frameIndex: Int     // First frame index in this chunk
    public let isFinal: Bool       // Last chunk flag
    public let elapsedTime: Double? // Wall-clock seconds (nil if not tracked)
    public let textTokens: [Int32] // Text tokens for this chunk (PersonaPlex streaming)
}
```

**Note on `textTokens`**: In `PersonaPlexModel.respondStream()`, each non-final chunk contains the text tokens generated during that chunk. The final chunk contains all text tokens from the entire generation. For non-PersonaPlex streams, this field defaults to empty.

### TranscriptionResult

Result of speech recognition including detected language:

```swift
public struct TranscriptionResult: Sendable {
    public let text: String
    public let language: String?          // e.g. "english", "russian"
    public let confidence: Float          // 0.0-1.0, 0.0 when the model does not provide it
    public let words: [WordConfidence]?   // per-word detail, nil when the model does not provide it
}

public struct WordConfidence: Sendable {
    public let word: String
    public let confidence: Float          // 0.0-1.0
    public let startTime: Double?         // seconds from the start of the audio, nil if unavailable
    public let endTime: Double?
}
```

`startTime` and `endTime` are emission-aligned, not forced-aligned: they mark the encoder frame at which the decoder emitted the word's tokens, and an offline encoder can stamp a word before its onset. `ParakeetASRModel` fills them; other models leave them nil. Use `Qwen3ForcedAligner` when you need acoustic word boundaries.

### SpeechSegment

Time segment where speech was detected, returned by `VoiceActivityDetectionModel`:

```swift
public struct SpeechSegment: Sendable {
    public let startTime: Float    // seconds
    public let endTime: Float      // seconds
    public var duration: Float     // computed: endTime - startTime
}
```

### AlignedWord

Word with timestamps, returned by `ForcedAlignmentModel`:

```swift
public struct AlignedWord: Sendable {
    public let text: String
    public let startTime: Float    // seconds
    public let endTime: Float      // seconds
}
```

### DiarizedSegment

Speech segment with speaker identity, returned by `SpeakerDiarizationModel`:

```swift
public struct DiarizedSegment: Sendable {
    public let startTime: Float    // seconds
    public let endTime: Float      // seconds
    public let speakerId: Int      // 0-based speaker identifier
    public var duration: Float     // computed: endTime - startTime
}
```

## Usage

### Generic TTS Function

```swift
import AudioCommon

func synthesizeAny(
    _ model: any SpeechGenerationModel,
    text: String,
    language: String? = nil
) async throws -> [Float] {
    try await model.generate(text: text, language: language)
}

// Works with any TTS model:
let qwen = try await Qwen3TTSModel.fromPretrained()
let cosy = try await CosyVoiceTTSModel.fromPretrained()
let vox = try await VoxCPM2TTSModel.fromPretrained()

let audio1 = try await synthesizeAny(qwen, text: "Hello")
let audio2 = try await synthesizeAny(cosy, text: "Hello")
let audio3 = try await synthesizeAny(vox, text: "Hello")
```

### Generic Streaming

```swift
func streamAny(
    _ model: any SpeechGenerationModel,
    text: String
) -> AsyncThrowingStream<AudioChunk, Error> {
    model.generateStream(text: text, language: nil)
}
```

### Existential Collections

```swift
let ttsModels: [any SpeechGenerationModel] = [qwen, cosy, vox]

for model in ttsModels {
    let audio = try await model.generate(text: "Hello", language: "english")
    print("Generated \(audio.count) samples at \(model.sampleRate) Hz")
}
```

## Module Structure

```
Sources/
├── AudioCommon/               Shared types, protocols, utilities
│   ├── Protocols.swift        AudioChunk, recognition updates, aligned/diarized segments, 13 protocols
│   ├── AudioModelError.swift  Unified error type for all model operations
│   ├── Logging.swift          Centralized os.Logger instances (AudioLog)
│   ├── AudioFileLoader.swift  WAV/audio file loading
│   ├── AudioFileStream.swift  Bounded pull decoding, resampling, channel routing
│   ├── WAVWriter.swift        WAV file writing
│   ├── HuggingFaceDownloader.swift  Safetensors / asset download from HF Hub
│   ├── Tokenizer.swift        BPE tokenizer
│   └── SentencePieceModel.swift  Shared SentencePiece `.model` protobuf reader (used by OmnilingualASR + PersonaPlex)
│
├── MLXCommon/                 MLX-specific utilities (quantised layers, weight loading, SDPA)
│   ├── WeightLoading.swift    Apply safetensors tensors to MLXNN modules
│   ├── QuantizedMLP.swift     Shared 4-bit SwiGLU MLP building block
│   ├── PreQuantizedEmbedding.swift  4-bit packed embedding table
│   ├── SDPA.swift             `SDPA.multiHead` / `SDPA.attendAndMerge` helpers used by every MLX attention module
│   └── MetalBudget.swift      GPU memory pinning / budget helpers
│
├── Qwen3ASR/                  Speech-to-text (ASR + Forced Aligner)
│   ├── Qwen3ASR.swift         Qwen3ASRModel: SpeechRecognitionModel
│   ├── ForcedAligner.swift    Qwen3ForcedAligner: ForcedAlignmentModel
│   ├── Qwen3ASR+Protocols.swift
│   └── ForcedAligner+Protocols.swift
│
├── WhisperASR/                Speech-to-text (Whisper Large-v3 Turbo CoreML)
│   ├── WhisperASR.swift       WhisperASRModel: SpeechRecognitionModel
│   ├── WhisperCoreMLRuntime.swift  Native CoreML mel/encoder/decoder runtime
│   └── WhisperByteLevelTokenizer.swift  Whisper byte-level BPE decoder
│
├── OmnilingualASR/            Speech-to-text (Meta wav2vec2 + CTC, 1,672 languages)
│   ├── OmnilingualASR.swift   OmnilingualASRModel: SpeechRecognitionModel (CoreML backend, 300M)
│   ├── Configuration.swift    Decodes published `config.json` (5 s / 10 s window variants)
│   ├── SentencePieceVocabulary.swift  Decoder built on `AudioCommon.SentencePieceModel`
│   ├── CTCGreedyDecoder.swift Argmax + consecutive-duplicate collapse
│   └── MLX/
│       ├── OmnilingualMLXModel.swift      OmnilingualASRMLXModel: SpeechRecognitionModel (MLX backend)
│       ├── OmnilingualMLXConfig.swift     Variant table (300M / 1B / 3B / 7B)
│       ├── Wav2Vec2Frontend.swift         CNN feature extractor + weight-normed pos encoder
│       ├── Wav2Vec2EncoderLayer.swift     Pre-norm transformer layer (quantised SA + FFN)
│       ├── Wav2Vec2Encoder.swift          Stack + final layer norm + CTC head
│       └── OmnilingualMLXWeightLoader.swift  Fuses PyTorch weight_norm at load time
│
├── Qwen3TTS/                  Text-to-speech (Talker + Code Predictor + Mimi)
│   ├── Qwen3TTS.swift         Qwen3TTSModel: SpeechGenerationModel
│   └── Qwen3TTS+Protocols.swift
│
├── CosyVoiceTTS/              Text-to-speech (LLM + DiT + HiFi-GAN)
│   ├── CosyVoiceTTS.swift     CosyVoiceTTSModel: SpeechGenerationModel
│   └── CosyVoiceTTS+Protocols.swift
│
├── VoxCPM2TTS/                Text-to-speech (MiniCPM-4 + LocEnc + LocDiT + AudioVAE V2)
│   ├── VoxCPM2TTS.swift       VoxCPM2TTSModel: SpeechGenerationModel
│   ├── MiniCPM4.swift         MiniCPM-4 backbone, LocEnc, LocDiT, UnifiedCFM
│   ├── AudioVAE.swift         AudioVAE V2 encode/decode
│   └── Configuration.swift    ModelArgs / config decoding for VoxCPM2 snapshots
│
├── IndexTTS2TTS/              IndexTTS2 voice cloning (reference conditioning + synthesis)
├── F5TTS/                     F5-TTS voice cloning (DiT flow + Vocos)
├── HiggsTTS/                  Higgs TTS 3 conversational TTS + cloning (Qwen3 + fused codebooks + codec)
│
├── PersonaPlex/               Speech-to-speech (Temporal + Depformer + Mimi)
│   ├── PersonaPlex.swift      PersonaPlexModel: SpeechToSpeechModel
│   └── PersonaPlex+Protocols.swift
│
├── SpeechVAD/                 VAD, end-of-turn detection, diarization, speaker embedding
│   ├── SpeechVAD.swift        PyannoteVADModel: VoiceActivityDetectionModel
│   ├── SpeechVAD+Protocols.swift  Protocol conformances
│   ├── SileroVAD.swift        SileroVADModel: VoiceActivityDetectionModel, StreamingVADProvider
│   ├── SileroModel.swift      Silero VAD streaming network (STFT + encoder + LSTM)
│   ├── StreamingVADProcessor.swift  Event-driven streaming wrapper
│   ├── SmartTurn.swift        SmartTurnModel: TurnCompletionProvider (Smart Turn v3.2 end-of-turn classifier, CoreML)
│   ├── DiarizationPipeline.swift  PyannoteDiarizationPipeline: SpeakerDiarizationModel, SpeakerExtractionCapable
│   ├── DiarizationHelpers.swift   Shared helpers (merge, compact IDs, resample)
│   ├── SortformerDiarizer.swift   SortformerDiarizer: SpeakerDiarizationModel (CoreML)
│   ├── WeSpeaker.swift        WeSpeakerModel: SpeakerEmbeddingModel
│   ├── ReDimNet2Speaker.swift ReDimNet2SpeakerModel: persistent named voice identity (CoreML)
│   └── PowersetDecoder.swift  7-class powerset → per-speaker probabilities
│
├── SpeechCore/                Voice pipeline (wraps speech-core C++ engine)
│   └── VoicePipeline.swift    VoicePipeline: bridges STT/TTS/VAD to C pipeline
│
├── AudioCLILib/               CLI commands and utilities (library)
└── AudioCLI/                  Thin launcher (main.swift → AudioCLILib)
```

### Dependencies

```
AudioCommon  ← Qwen3ASR         ─┐
             ← WhisperASR       │
             ← Qwen3TTS         │
             ← CosyVoiceTTS     │
             ← VoxCPM2TTS       │
             ← IndexTTS2TTS     │
             ← KokoroTTS        ├── AudioCLILib ── AudioCLI (executable)
             ← ParakeetASR      │
             ← ParakeetStreamingASR │
             ← OmnilingualASR   │  (CoreML + MLX backends)
             ← PersonaPlex      │
             ← SpeechVAD       ─┘
             ← SpeechCore (CSpeechCore xcframework + AudioCommon)

MLXCommon  ← Qwen3ASR, Qwen3TTS, Qwen3Chat, CosyVoiceTTS, VoxCPM2TTS,
              PersonaPlex, SpeechVAD, OmnilingualASR (MLX backend)
```

Each model target depends only on `AudioCommon` and MLX. No cross-dependencies between model targets. `SpeechCore` depends on `AudioCommon` for protocols and the `CSpeechCore` binary target for the C++ pipeline engine.

## Thread Safety

All model classes are **not thread-safe** by design. ML inference is inherently sequential on a shared GPU, and MLX's `Module` system does not support actor isolation. Adding synchronization primitives would introduce overhead for a scenario no caller exercises.

**Not thread-safe** (create separate instances for concurrent use):
- `Qwen3ASRModel`, `StreamingASR`
- `Qwen3TTSModel`
- `CosyVoiceTTSModel`
- `VoxCPM2TTSModel`
- `IndexTTS2TTSModel`
- `PersonaPlexModel`
- `OmnilingualASRModel` (CoreML), `OmnilingualASRMLXModel` (MLX)
- `ParakeetASRModel`, `ParakeetStreamingASRModel`, `NemotronStreamingASRModel`
- `SileroVADModel`, `StreamingVADProcessor`, `PyannoteVADModel`
- `PyannoteDiarizationPipeline` (aliased as `DiarizationPipeline`), `SortformerDiarizer`

**Thread-safe** (all `let` properties, pure computation):
- `WeSpeakerModel`

**Sendable config types** — The following value types conform to `Sendable` and can be safely passed across concurrency boundaries:
`SegmentationConfig`, `VADConfig`, `DiarizationConfig`, `VADPipeline`, `Qwen3AudioEncoderConfig`, `Qwen3ASRTokens`, `SlottedText`, `TextChunker`

## OpenAI-Compatible HTTP Audio

The `speech-server` binary exposes the OpenAI audio request shapes alongside
its native REST endpoints:

- `POST /v1/audio/transcriptions` accepts multipart audio and returns an
  OpenAI transcription response.
- `POST /v1/audio/speech` accepts JSON with `model`, `input`, and `voice` and
  returns synthesized audio.

```bash
curl http://localhost:8080/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"tts-1","voice":"alloy","input":"Hello","response_format":"wav"}' \
  -o output.wav
```

The OpenAI model aliases `tts-1`, `tts-1-hd`, `gpt-4o-mini-tts`, and
`gpt-4o-mini-tts-2025-12-15` select Kokoro. Native registry aliases select
their corresponding TTS engine. Generic OpenAI voice names use the selected
engine's default voice, while native Kokoro and Qwen3-TTS voice names pass
through. `speed` is supported by Kokoro; the optional `language` extension
defaults to English.

The endpoint defaults to WAV and also supports headerless 24 kHz mono PCM16
little-endian output with `response_format: "pcm"`. Compressed OpenAI formats
are rejected explicitly rather than returning PCM under an incorrect media
type.

## Error Handling

### Realtime WebSocket errors

The `/v1/realtime` websocket returns OpenAI-style error events instead of
closing the connection for recoverable request or model-processing failures:

```json
{
  "type": "error",
  "error": {
    "type": "server_error",
    "message": "Realtime event 'response.create' failed: ...",
    "event_type": "response.create"
  }
}
```

Client request errors use `invalid_request_error`. Failures thrown while
processing a valid realtime event use `server_error` and include `event_type`
so clients can associate the error with the message that failed.

Long-running realtime model loads and generations emit lightweight
`realtime.keepalive` events and websocket pong control frames periodically
while no model output is ready. Clients can ignore these events or treat them
as cold-start activity indicators.

### Realtime server VAD

Realtime sessions use explicit `input_audio_buffer.commit` by default. Enable
automatic turn detection with a session update:

```json
{
  "type": "session.update",
  "session": {
    "turn_detection": {
      "type": "server_vad",
      "threshold": 0.5,
      "prefix_padding_ms": 300,
      "silence_duration_ms": 500,
      "max_turn_duration_ms": 120000
    }
  }
}
```

The server then emits `input_audio_buffer.speech_started`,
`input_audio_buffer.speech_stopped`, and `input_audio_buffer.committed` before
the normal transcription events. Silence retains only the configured pre-roll
and detection history, and ASR runs only after a speech turn closes. Set
`turn_detection` to `null` or `{"type":"none"}` to restore manual commits.

### AudioModelError

Unified error type in `AudioCommon` for cross-module error reporting:

| Case | Fields | When |
|------|--------|------|
| `modelLoadFailed` | `modelId`, `reason`, `underlying?` | Model download or initialization fails |
| `weightLoadingFailed` | `path`, `underlying?` | Safetensors file cannot be read |
| `inferenceFailed` | `operation`, `reason` | Generation or decoding step fails |
| `invalidConfiguration` | `model`, `reason` | Config values are incompatible |
| `voiceNotFound` | `voice`, `searchPath` | Voice preset file missing |

Each case produces a human-readable `errorDescription` with full context including underlying errors.

### Per-module errors

Modules may also define their own error types for domain-specific failures:
- `TTSError` (Qwen3TTS) — tokenizer and language errors
- `CosyVoiceTTSError` (CosyVoiceTTS) — load, download, input, generation errors
- `DownloadError` (AudioCommon) — HuggingFace download failures

## Logging

Centralized structured logging via `os.Logger` (Apple's unified logging system):

```swift
import AudioCommon

// Available loggers:
AudioLog.modelLoading  // Weight loading, initialization, voice preset errors
AudioLog.inference     // Generation, decoding, pipeline steps
AudioLog.download      // HuggingFace downloads, cache operations
```

All loggers use subsystem `com.qwen3speech`. Messages are visible in Console.app and `log stream`.

Used in:
- `PersonaPlexModel` — voice preset loading failures (`.warning`)
- `HuggingFaceDownloader` — directory listing errors (`.debug`)

## Design Decisions

1. **`AnyObject` constraint** — All protocols require reference semantics since ML models hold large weight buffers
2. **Optional `language`** — Protocol methods use `String?` to allow model-specific defaults (Qwen3 defaults to "english", CosyVoice to "english")
3. **Optional `elapsedTime`** — `AudioChunk.elapsedTime` is `Double?` because not all models track wall-clock time (e.g. CosyVoice)
4. **No `ModelLoadable`** — Each model has different loading parameters (TTS needs `tokenizerModelId`, PersonaPlex needs voice presets), so loading stays on concrete types
5. **Unified `AudioChunk`** — All streaming methods return the shared `AudioChunk` type directly. The previous per-model chunk types (`TTSAudioChunk`, `CosyVoiceAudioChunk`, `PersonaPlexAudioChunk`) were removed
6. **Separate `ForcedAlignmentModel`** — Distinct from `SpeechRecognitionModel` because input/output differ (audio+text → timestamps vs audio → text)
7. **Document-only thread safety** — No locks or actors; document the single-threaded contract instead. This matches standard ML library practice (PyTorch, Core ML)
8. **Sendable on value types** — Config structs with only primitive fields get `Sendable` so they can cross `Task` boundaries without warnings