audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead. Runs on Windows, Linux, and macOS, with support for NVIDIA, AMD, Apple Silicon, and CPU-only machines.

Lost in a sea of audio model names, papers, and codebases? Explore 100+ audio models in the Audio Model Architecture Atlas to see how they're built, what they share, and where they differ, whether you're learning or building your own.

[!IMPORTANT]

2026-09-23 - Release v0.8.2: Day-zero support for NVIDIA Nemotron 3 Diarization, with streaming and batched inference. AuK and LiveAvatar are now available in the WebUI.

Arena UI: The new Arena tab makes it easier to compare local models side by side for TTS, voice conversion, and ASR. Use one shared input, queue multiple models or GGUF variants, then review outputs with metrics!

CUDA performance headline: multiple TTS paths already run 1.8x to up to 8x faster than their Python reference paths while cutting end-to-end latency by 45%-85%.

GGUF performance: all released model families support GGUF loading, and tested Q8 packages can run up to 1.53x faster while reducing peak VRAM by up to about 37% on routes such as Higgs Audio, Fish Audio, and Voxtral. See the GGUF guide for support status and the Q8 performance report for 16-bit vs Q8 measurements.

Production deployment example: Try Fun-ASR-Nano and SenseVoice with audio.cpp on the FunASR platform!

VibeVoice 1.5B: generates a 93.9-minute podcast in 18.2 minutes with 10 diffusion steps and without quantization, running about 5.15x faster than real time.

Supertonic 3: generates about 10 hours of audio in 3 minutes on RTX5090. Up to 200x+ real-time on CUDA, 6x+ real-time on CPU, and 47 ms TTFT in CUDA streaming mode. Demo: 10 hours of audio generated in 3 minutes.

Real-world ASR win: In TranscrIA benchmark on messy French meeting audio, audio.cpp’s Nemotron 3.5 ASR matched the same WER as other implementations while using about 1/4 of the wall time.

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CUDA, HIP/ROCm, Vulkan, Metal, and CPU backends behind shared CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths. Model-based utilities need separately downloaded weights; see downloads and usage.

The goal of the framework is to provide highly optimized, reusable building blocks for audio-related models, so new model integrations can be brought up faster, shared components can be improved once and benefit many families, and real end-to-end inference paths can stay efficient, maintainable, and portable.

audio.cpp would not be moving this quickly without generous contributors bringing in real fixes, new capabilities, and careful polish. See CONTRIBUTING.md for how to contribute and for a shout-out to the people already helping shape the project.

[!TIP] Contribution focus: the most helpful contributions right now are improvements to the UI, API server, and pipeline/workflow subsystems. These areas make the existing model surface easier to use, serve, compose, and validate. See CONTRIBUTING.md for more details.

New model PRs: before starting a new model port, please check the supported model table because several families are already implemented or under testing and read the New Model PRs section in CONTRIBUTING.md. New ports should start under the community models surface, where review is lighter than core models but still needs reproducible validation.

News

[!IMPORTANT] 2026-09-15 - Release 0.8.0: Since v0.7.0, audio.cpp has added YuE2 song generation, SheetSage2 audio-to-score transcription, BreezeTTS 2, CosyVoice3, Kokoro 82M, Moonshine Streaming ASR, Niagara ASR, and VibeVoice ASR Streaming 7B. The community lineup also grows with Audio8 ASR/TTS, Echo-TTS, Chatterbox Turbo, MiraTTS, Sopro, Soprano, sanoTTS, VoxCPM1, VibeASR, and Sortformer v2.1, alongside Irodori v4.1 Anime support. This brings audio.cpp to 80+ model families and 120+ model variants! This release also brings improvements to performance, stability, and usability across the project. Thanks to the community for the contributions, testing, and feedback!

Optional server frontend modules add MP3 support, HTTPS listeners, and more customization for server integrations.

Thanks to @DrewThomasson for the Colab UI, @christopherthompson81 for the C ABI, and @XsquirrelC/VibeASR for the official VibeASR port, extending audio.cpp with support for BitNet models!

2026-08-26 - Release 0.7: This release adds MiniMax Music 3, MagpieTTS, PersonaPlex, MeanVC2, AudioSR, ControlFoley, FireRedTTS3, FireRedAudio, MiDashengLM-Gen, F5-TTS/Habibi, Granite Speech 5.0 TurboCTC, MMS Forced Aligner, and MOSS-VoiceGenerator, plus DotTTS Edit and ACE-Step 1.5 XL variants, bringing audio.cpp to 62 total model families and 85+ model variants! It also introduces the new Arena UI for side-by-side TTS, voice-conversion, and ASR comparison with shared inputs, queued runs, metrics, and result sorting.

2026-08-13 - Release 0.6: This release adds 5 new model families - DotTTS, NeuTTS, MuScriptor, MiniMax-H3, and SenseVoice - bringing audio.cpp to 49 total model families and 70+ model variants, alongside the new native WebUI from @mirek190, expanded GGUF packaging, and more shared framework runtime pieces.

2026-07-31 - Release 0.5: audio.cpp reaches 44 model families with 9 new additions, early HIP/ROCm support for AMD GPUs, Nix ROCm/HIP build support, Metal optimizations with tested VoxCPM2 runs up to 2.56x faster on Apple Silicon, and a major GGUF-first WebUI/package-spec usability pass.

2026-06-25 to 2026-07-23 (release 0.1 to 0.4): audio.cpp grew from the first released model wave into broad TTS, ASR, music generation, source separation, VAD, diarization, codec, and voice-conversion coverage, with VibeVoice 1.5B/7B, LoRA adapter loading, initial streaming support, and major CUDA Conv1DTransp speedups.

Supported Models

Task tags: TTS text to speech, Clone voice cloning, VC voice conversion, S2S speech-to-speech, ASR speech recognition, Align forced alignment, VAD voice activity detection, Diar speaker diarization, Codec audio codec, Sep source separation, MIDI audio-to-symbolic MIDI/events, Music music/song generation, SFX sound effects, Video video generation, Edit audio/music editing, Design voice design, Dialogue multi-speaker dialogue TTS, Ctrl TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.

Runtime tags summarize the supported loading paths. GGUF package precision varies by model and release; check the audio.cpp GGUF repo or docs/gguf.md for the exact package list. Bundled means the tiny runtime asset ships under assets/framework/models and needs no separate model download. Stream means the family exposes a streaming server/session path.

Model weights keep the license of their original release, which is separate from audio.cpp's own license. docs/model_licenses.md lists that license per family, and whether it allows commercial use.

Speech Generation And Conversation

Speech Recognition And Analysis

Audio Conversion And Processing

Music, Media, And Editing

Some model families in the supported table started as outside contributions before being promoted into the core release surface. Thanks to Mirek @mirek190 for BS-RoFormer, @justinjohn0306 for MOSS-TTS-Local, and @LauraGPT from the official FunASR team for Fun-ASR-Nano.

Community Models

Community model ports live under community_models to make the ownership boundary clear while keeping them available through the normal audio.cpp CLI and server paths. Some community-contributed models graduate into the core model tree when they become part of the main release surface. Huge thanks to the contributors who bring these models in, test them, and keep pushing the framework into new territory. See docs/community_models/models.md for community-model expectations and current entries.

Prebuilt Binaries

Official packages for every release are attached to the Releases page:

The Windows CUDA packages ship the CUDA runtime in a separate cudart archive. Extract it next to the binaries so the CUDA backend can load ggml-cuda.dll and the CUDA runtime DLLs (cudart, cuBLAS, cuFFT).

• Windows (HIP/ROCm, AMD GPUs): community-maintained packages with the ROCm runtime bundled, so no HIP SDK installation is required. Published from @IIIIIllllIIIIIlllll's fork Releases in two tracks: ROCm 6.4 (full coverage incl. RX 7600 / gfx1102) and ROCm 7.1 (recommended for RDNA4). Version numbers follow the upstream releases; see docs/build/windows-hip-distribution.md for details.

Homebrew Install

On macOS, audio.cpp can be installed from the Homebrew tap:

brew tap 0xShug0/audio-cpp
brew trust 0xShug0/audio-cpp
brew install audio-cpp

WebUI

audiocpp_server includes an embedded SvelteKit/TypeScript WebUI for running local TTS, cloning, ASR, generation, conversion, separation, VAD, diarization, and alignment workflows. The production UI is compiled into the server binary, so using it requires neither Python nor separate frontend files:

audiocpp_server --ui --ui-management --backend cuda

Open http://127.0.0.1:8080. --ui-management enables catalog browsing, downloads, temporary uploads, and dynamic model switching, so this is the easiest way to try audio.cpp without writing a server config first. For a locked-down config-driven instance, start with --ui and a server config instead; in that mode the UI only offers models declared by the config unless management is enabled.

For Docker UI downloads, mount a writable models directory at /app/models and use --ui-management. See docker.md.

The native UI also exposes background model download/preparation, long-text split-and-merge synthesis, a browser-local saved voice library, microphone recording, and near-live ASR input. Some model preparation jobs invoke the repository's Python model manager because those packages require Hugging Face download or checkpoint conversion; model inference and the embedded UI remain Python-free. See webui/README.md for launch commands, model notes, and frontend development instructions.

Docker

Docker CUDA and CPU images are available for both CLI and server use. See docker.md for available images, build commands and working Docker examples.

Model Manager and GGUF Downloads

Use tools/model_manager_v2.py for normal model downloads. It reads model_specs/*.json and installs the default package for each family, preferring ready-to-use GGUF packages when they are available.

Native builds configured with -DAUDIOCPP_BUILD_NATIVE_MODEL_MANAGER=ON also provide audiocpp_model_manager, a standalone headless frontend over the same reusable C++ package-management library used by the server and embedded WebUI. It supports list, info, install, clean, and remove without starting audiocpp_server; this is the preferred native path for CLI, Docker, remote provisioning, and other scripted environments. The Python v2 manager remains available as an alternative during migration. The native path uses bundled TLS by default and does not require libcurl.

audiocpp_model_manager list
audiocpp_model_manager install qwen3_asr_0_6b_q8_0 --models-dir models

The old safetensors/converter catalog has been renamed to tools/model_manager_deprecated.py. Use it only for legacy model layouts that have not moved to spec-backed GGUF packages yet.

GGUF downloads:

• Released model packages: audio-cpp/audio.cpp-gguf
• Community model package: mirek190/audio.cpp

See the Model Manager guide for model-manager usage and package notes.

Build yourself

scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli

cmake -S . -B build/debug -DCMAKE_BUILD_TYPE=Debug -DAUDIOCPP_MODEL_SET=custom -DAUDIOCPP_MODELS=qwen3_tts,pocket_tts,qwen3_asr
cmake --build build/debug --target audiocpp_cli -j 8

scripts/build_linux.sh --backend cuda --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend vulkan --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cpu --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --cuda-arch native --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend cuda --cuda-arch "86;89" --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --cuda-arch native --ccache --target audiocpp_cli

scripts/build_linux.sh --backend cuda --model-set full --target audiocpp_cli
scripts/build_linux.sh --backend cuda --model-set custom --models qwen3_tts,pocket_tts,qwen3_asr --target audiocpp_cli
scripts/build_linux.sh --backend cpu --model-set core --target audiocpp_cli

scripts/build_linux.sh --backend cuda --native-cpu OFF --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --deployment-build --target audiocpp_cli --target audiocpp_server

scripts/build_linux.sh --backend cuda --native-model-manager --target audiocpp_server

powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\scripts\build_windows.ps1

.\scripts\build_windows.ps1 -Preset windows-cuda-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-vulkan-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Preset windows-cpu-release -Target audiocpp_cli
.\scripts\build_windows.ps1 -Target audiocpp_server -Jobs 16
.\scripts\build_windows.ps1 -Preset windows-cuda-release -ModelSet custom -Models "qwen3_tts,pocket_tts,qwen3_asr" -Target audiocpp_cli

scripts\build_windows.cmd

.\scripts\build_windows.ps1 -DeploymentBuild -Target audiocpp_cli

.\scripts\build_windows.ps1 -NativeModelManager -Target audiocpp_server

cmake -S . -B build/macos-cpu-release \
  -DCMAKE_BUILD_TYPE=Release \
  -DENGINE_ENABLE_CUDA=OFF \
  -DENGINE_ENABLE_VULKAN=OFF \
  -DENGINE_ENABLE_METAL=OFF \
  -DENGINE_ENABLE_OPENMP=OFF \
  -DGGML_OPENMP=OFF
cmake --build build/macos-cpu-release \
  --parallel "$(sysctl -n hw.logicalcpu)" \
  --target audiocpp_cli audiocpp_server audiocpp_gguf

build/macos-cpu-release/bin/audiocpp_cli --list-devices

scripts/build_metal.sh --target audiocpp_cli

scripts/build_metal.sh --target audiocpp_server
scripts/build_metal.sh --build-type Release --archs arm64 --target audiocpp_cli
scripts/build_metal.sh --model-set custom --models qwen3_tts,pocket_tts --target audiocpp_cli
scripts/build_metal.sh --with-tests --target audio_dsp_test
scripts/build_metal.sh --openmp auto --target audiocpp_cli
scripts/build_metal.sh --native-cpu OFF --target audiocpp_cli
scripts/build_metal.sh --deployment-build --target audiocpp_cli

build/macos-metal-release/bin/audiocpp_cli

scripts/build_linux.sh --backend hip --target audiocpp_cli --target audiocpp_server
scripts/build_linux.sh --backend hip --gpu-targets "gfx1100;gfx1103" --target audiocpp_cli

cmake -S . -B build_hip \
  -DENGINE_ENABLE_HIP=ON \
  -DGPU_TARGETS=gfx1151 \
  -DCMAKE_C_COMPILER="$(hipconfig -l)/clang" \
  -DCMAKE_CXX_COMPILER="$(hipconfig -l)/clang++" \
  -DCMAKE_BUILD_TYPE=Release
cmake --build build_hip -j$(nproc)

powershell -ExecutionPolicy Bypass -File scripts\build_windows_hip.ps1

git -c 'url.https://github.com/.insteadOf=git@github.com:' submodule update --init external/audio.cpp-server-frontends

cmake -S . -B build/debug \
  -DCMAKE_BUILD_TYPE=Debug \
  -DAUDIOCPP_BUILD_SERVER_FRONTENDS=ON \
  -DAUDIOCPP_SERVER_FRONTENDS_DIR=external/audio.cpp-server-frontends \
  -DAUDIOCPP_SERVER_FRONTEND_MODULES="audio_decode;mp3_encode"
cmake --build build/debug --target audiocpp_server --parallel 8

Usage

[!IMPORTANT] Please read your model's documentation: open the model-docs index, choose your task category, then find your model. Also read the dedicated CLI usage guide and server guide. The examples below are quick starts; model requirements, supported options, and setup details vary.

CLI

The main CLI binary is:

build/bin/audiocpp_cli

High-level command shape:

audiocpp_cli --task <task> --model <path> [--family <family>] [--backend <backend>] [--mode <mode>] [options]

Core selectors:

• --task vad|asr|diar|sep|gen|tts|clon|vc|s2s|align|vdes|spk|svc
• --model <path>
• --family <name> optionally narrows model-loader selection when a model path could match more than one family
• --backend cpu|cuda|vulkan|metal|best
• --mode offline|streaming; streaming is available for models whose docs list streaming support

Common interface options:

• --load-option key=value passes model-load options, such as PocketTTS language selection
• --session-option key=value passes session/runtime options, such as backend-specific weight controls
• --request-option key=value passes per-request model options
• --config <id> selects a discovered config asset
• --weight <id> selects a discovered weight asset
• --device <n> selects the backend device
• --threads <n> sets backend and OpenMP worker threads

Examples:

Text-to-speech:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/model \
  --backend cuda \
  --text "audio.cpp is running PocketTTS locally." \
  --voice-ref assets/resources/sample.wav \
  --out build/out/pocket_tts.wav

PocketTTS with another language and a built-in voice:

build/bin/audiocpp_cli \
  --task tts \
  --family pocket_tts \
  --model /path/to/models/pocket-tts \
  --backend cuda \
  --load-option language=spanish \
  --text "Hola, esta es una prueba corta de Pocket TTS." \
  --voice-id alba \
  --out build/out/pocket_tts_spanish.wav

ASR:

build/bin/audiocpp_cli \
  --task asr \
  --family qwen3_asr \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/sample_16k.wav

Voice conversion:

build/bin/audiocpp_cli \
  --task vc \
  --family seed_vc \
  --model /path/to/model \
  --backend cuda \
  --audio assets/resources/a.wav \
  --voice-ref assets/resources/b.wav \
  --out build/out/seed_vc.wav

Useful CLI features:

• --help with --task shows task-oriented help
• --help with --model <path> and optional --family <family> shows model-owned request, session, and load options
• --inspect prints discovered configs, weights, and capabilities
• --list-loaders prints registered model families (--json for the machine-readable contract)
• python tools/model_manager_v2.py list --json prints installable packages from model_specs/*.json
• --batch-text-file <txt> runs one offline request per non-empty line
• --batch-text-dir <dir> runs one offline request per .txt, .md, or .json file, normalizing each file as one paragraph
• --batch-audio-dir <dir> runs one offline request per .wav
• --audio-chunk-mode auto lets ASR/alignment models choose their safe long-audio policy; expert users can override with fixed, vad, or none where supported
• --request-sequence <json> runs a multi-request offline session
• --batch-merge-audio none|concat controls batch audio merge behavior
• --batch-manifest-out <json> writes a batch output manifest
• --metrics prints compact offline wall time, audio duration, RTF, realtime speed, sample rate, and channel metrics
• Use --request-sequence <json> --metrics for per-request metrics from one long-lived offline session
• --pipeline <json> runs a workflow instead of a raw task
• --list-pipelines prints registered workflows
• --workflow-input key=value overrides pipeline inputs
• --log streams framework logs to stdout
• --log-file <path> streams framework logs to a file in real time
• --segments-out, --turns-out, and --words-out write structured JSON outputs
• --vad-chunks-out writes offline VAD-based chunk windows; tune them with --vad-chunk-max-seconds, --vad-chunk-merge-gap-seconds, and --vad-chunk-padding-seconds

Server

The server binary is:

build/bin/audiocpp_server

Build:

cmake --build build -j$(nproc) --target audiocpp_server

Create a config file with your own model paths:

cat > server.json <<'JSON'
{
  "host": "127.0.0.1",
  "port": 8080,
  "backend": "cuda",
  "device": 0,
  "threads": 1,
  "lazy_load": true,
  "models": [
    {
      "id": "pocket-tts",
      "family": "pocket_tts",
      "path": "/path/to/models/pocket-tts",
      "task": "tts",
      "mode": "offline",
      "load_options": {
        "language": "english"
      },
      "session_options": {
        "language": "english"
      }
    },
    {
      "id": "qwen3-asr",
      "family": "qwen3_asr",
      "path": "/path/to/models/Qwen3-ASR-0.6B",
      "task": "asr",
      "mode": "offline"
    }
  ]
}
JSON

Set "lazy_load": true to register configured model ids at startup while loading each model only on first use. Use per-model "lazy": true or "lazy": false to override that default.

Set top-level "backend" to "cuda", "cpu", "vulkan", "metal", or "hip". CUDA is the optimized path for audio.cpp; CPU, Vulkan, Metal, and HIP are intended for portability and testing when the binary is built with that backend, but performance and model coverage may be lower.

[!WARNING] Lazy loading does not unload models after a request. Once a model is first used, the server keeps that model and session in memory for reuse until the server exits, unless max_loaded_models limits residency.

Set top-level "max_loaded_models" (or start with --max-loaded-models <n>) to bound how many models stay resident in memory at once: loading one more past the limit first unloads the least recently used idle model, and 1 enforces a single loaded model at a time. The default 0 keeps every used model in memory. See app/server/README.md for details.

Start:

build/bin/audiocpp_server --config server.json

The server exposes:

• GET /health
• GET /v1/models
• POST /v1/audio/speech
• POST /v1/audio/transcriptions
• POST /v1/audio/transcriptions/details
• POST /v1/audio/alignments
• POST /v1/tasks/run

More server examples are in app/server/README.md.

Pipelines

Pipelines are an experimental JSON workflow feature for chaining multiple model and audio-processing steps behind one CLI command. A pipeline can define default inputs, let users override them with --workflow-input key=value, split long media into model-sized chunks, merge text or audio outputs back together, write intermediate artifacts under --out-dir, and copy the declared final_audio to --out.

This is the higher-level layer for production-style audio jobs: redubbing, batch cleanup, long-form narration, voice conversion, source-separation workflows, transcription-plus-alignment, and future workflows that combine translation, diarization, denoise, enhancement, or review steps as those model surfaces are wired into the framework.

The included same-language speech redub pipeline transcribes long speech in chunks with Qwen3 ASR, merges the transcript, then regenerates the speech in a target reference voice with Qwen3 TTS. The default test input assets/resources/speech.wav is about 418 seconds long and was generated from an 8,091-character speech text, so it exercises long-audio split and merge behavior rather than a short one-shot request:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Override the source speech or target voice without editing the JSON:

build/bin/audiocpp_cli \
  --pipeline assets/pipeline/speech_redub.json \
  --backend cuda \
  --workflow-input source_audio=/path/to/speech.wav \
  --workflow-input target_voice=/path/to/voice.wav \
  --workflow-input language=English \
  --out-dir build/out/speech_redub_pipeline \
  --out build/out/speech_redub_pipeline.wav

Tests

The repository includes both framework-level parity validation and app-level end-to-end path checks. At a high level, the flow is:

The main harness under tests/ is tests/warmbench.py. It is used for long-lived multi-request validation, parity checks against Python references, and performance-oriented session reuse scenarios. The tests/ tree also contains model-specific C++ and Python warmbench entrypoints that warmbench.py coordinates.

The main app-facing test tooling under tools/ is tools/audiocpp_cli/run_audiocpp_cli_path_tests.py. It drives audiocpp_cli through cataloged offline and streaming cases, verifies expected outputs such as audio or JSON artifacts, and is useful for checking real user-facing request paths rather than just lower-level model components. Streaming coverage is model-specific and applies to models documented with streaming support.

The Python-reference side of these tests usually requires more time-consuming setup than the C++ path because different models rely on different Python reference repos and dependency stacks. In practice, the framework-side tooling is fast to iterate on once models are installed, while Python parity runs often need extra environment preparation before they are ready.

Projects

Last update: 2026-09-23

Have a project using audio.cpp? Submit a PR or let me know, and I’ll be happy to add it here.

• AIRI Audio Server: A lightweight, zero-Python Node.js microservice wrapper providing OpenAI-compatible /v1/audio/speech (with real-time SSE streaming), /v1/audio/transcriptions (Citrinet/Whisper), dynamic voice catalog discovery, and serialized GPU queueing for audio.cpp.
• Project AIRI: An open-source, next-generation AI companion and desktop frontend featuring interactive Voice Studio curation, zero-shot voice cloning dropzones, and real-time audio pipeline integration powered by airi-audio-server and audio.cpp.
• TranscrIA is a self-hosted meeting transcription platform with diarization and local LLM correction. audio.cpp is integrated as a first-class STT engine in the product.
• Pocket TTS Browser Engine uses audio.cpp to bring fully local PocketTTS voices into Chrome and Edge through the browser TTS API.
• GuideAnts uses audio.cpp as the default local AI stack path for basic ASR and TTS, with planned reusable skills for audio.cpp scenarios and model configurations.
• audio.cpp-webui provides a full-task Python/Gradio WebUI for audio.cpp, focused on browser-based model downloads and common TTS, ASR, voice conversion, diarization, music, and audio workflows.
• audio.cpp-hub is a Web GUI for audio.cpp that packages model-oriented workflows around the native runtime.
• Delusion provide Pythonic, strongly typed wrapper classes around audio.cpp usage, including model download/cache helpers and typed request surfaces.
• AudioCppTray is a Windows tray management tool for audiocpp_server.exe, with start/stop/restart controls, notifications, log viewing, log rotation, and server configuration shortcuts.
• AudioCpp.NET provides .NET 10 bindings for audio.cpp behind a small versioned C ABI shim, shipped as managed, CPU runtime, and CUDA runtime NuGet packages for Windows and Linux.
• yovoice is a desktop voiceover app for macOS and Windows, built on audio.cpp with support for local text-to-speech, voice cloning, and emotion control.
• AudioCpp-Bindings provides .NET bindings over the C ABI merged in #530, an Avalonia desktop app for Linux, macOS and Windows that mirrors the web UI's seven workflows and model manager (with its interface strings imported from the web UI's own language files, so the two read alike in English, Italian, Polish, Russian and Simplified Chinese), and a C# reimplementation of the HTTP API that an existing client can be pointed at unchanged.
• Remiqora is an open-source (MIT) local music studio for Windows, macOS and Linux that runs YuE2-3B on audio.cpp's native server (plus MuScriptor for audio-to-MIDI) next to ACE-Step 1.5, swapping the two engines on a single GPU. It adds one-click Demucs stem splitting, a browser multitrack editor with an effects rack and WAV/MP3 export, and a LoRA training UI.

Performance Metrics

[!WARNING] These Python-relative numbers were measured for the initial release. Several model paths have improved substantially since then, so the figures below should be read as the original release baseline rather than the latest peak performance.

All performance metrics in this section were measured on Ubuntu with the CUDA backend on an NVIDIA GeForce RTX 5090. The Python-relative one-shot and long-lived-session comparisons come from direct framework/runtime API benchmark calls, not from audiocpp_cli; CLI path tests are separate and include app-layer request parsing, output writing, and other user-facing overhead.

Absolute RTF depends on the GPU and system setup, but the Python-relative speedups are real because audio.cpp and the matching Python reference paths were measured on the same CUDA setup.

audio.cpp already shows some genuinely exciting wins against the matching Python reference paths, especially on the TTS side, even when using the original model weights without quantization. The headline win is wall time: several TTS paths run 1.8x to up to 10x faster than Python while cutting end-to-end latency by 45%-90%.

• In one-shot runs, several TTS-family models already land far ahead of Python:
  • vevo2: 5.03x faster with 80.11% less wall time
  • pocket tts: 3.68x faster with 72.80% less wall time
  • miotts: 2.73x faster with 63.39% less wall time
  • moss_tts_local: 2.33x faster with 57.07% less wall time
  • qwen3 tts: 1.83x faster with 45.34% less wall time
  • vibevoice: 1.40x faster with 28.75% less wall time
• In long-lived-session runs, where the same loaded session serves multiple requests in sequence, the gains stay strong:
  • pocket tts: 3.22x faster with 68.91% less wall time
  • qwen3 tts: 2.74x faster with 63.47% less wall time
  • moss_tts_local: 2.66x faster with 62.35% less wall time
  • miotts: 2.28x faster with 56.22% less wall time
  • vibevoice: 1.77x faster with 43.55% less wall time
  • vevo2: 1.75x faster with 42.72% less wall time
• In long-form runs on the shared 6,026-character, 1,028-word passage, the strongest Python-relative wins still show up clearly:
  • pocket tts: 3.15x faster with 68.23% less wall time
  • qwen3 tts: 3.06x faster with 67.33% less wall time
  • vibevoice: 2.86x faster with 65.07% less wall time
  • vevo2: 1.77x faster with 43.51% less wall time
  • chatterbox: 1.58x faster with 36.83% less wall time
• These long-lived-session numbers are especially important for real applications, because they reflect the common case where model load, cached state, and reusable runtime setup are amortized across many requests.
• Bars below the 1.0x line are useful too: they spotlight exactly where more optimization work is still worth doing.

The figures report Python wall time / audio.cpp wall time. The 1.0x line means equal wall time; bars above 1.0x mean audio.cpp is faster than Python, and bars below 1.0x mean it is slower.

For TTS-family models, the measured one-shot RTF is:

For long-form TTS tests, each run uses the same 6,026-character, 1,028-word input text (vibevoice uses 106,310 chars, 18,052 words, 4 speakers). Rows are CUDA unless marked CPU. The measured RTF is:

Runtime Memory Options

Some models expose memory-saver session options such as ace_step.mem_saver=true, dramabox.mem_saver=true, heartmula.mem_saver=true, stable_audio.mem_saver=true, omnivoice.mem_saver=true, voxcpm2.mem_saver=true, and echo_tts.mem_saver=true. These options keep the default output path unchanged while reducing graph workspace VRAM or releasing staged graph/cache state after request phases; later requests may rebuild released graphs.

Precision/Quantization Support

Many model sessions expose quantization through --session-option <family>.weight_type=<mode>, and some families also expose more specific knobs such as ...conv_weight_type, ...talker_weight_type, or ...speech_decoder_weight_type. The exact supported modes are model-specific rather than global.

audio.cpp also supports standalone GGUF packages.

In practice, lower precision and quantized modes should be treated as model- and route-specific optimizations rather than universally safe defaults.

• Safety. Quantization may not be safe on every path even when a model parser accepts the option. For example, in our ACE-Step 1.5 checks, lower-precision runs could fail at runtime with ACE-Step planner masked decode found no valid token while higher-precision settings completed normally.

• Quality Drop. Output quality can drop a lot. In our VeVo2 checks, non-fp32 outputs showed noticeably weaker similarity to the fp32 reference under the repo's existing waveform and log-mel comparison metrics, and even output length could shift.

• Performance Gain. The performance gain may be minor relative to that quality risk. For example, q8_0 was faster than the default setting by only around 3.8% on Qwen3-TTS and around 3.6% on VeVo2. Other models may benefit more, but the tradeoff should be validated per model and per route rather than assumed.

• Memory Benefit. Lower precision and quantized weights can still be useful for reducing weight memory footprint and making larger models easier to fit within device limits. For example, in our Qwen3-TTS checks, switching from the default setting to q8_0 reduced peak RAM by about 3.7% and peak VRAM by about 25.0%. That benefit is real, but it should be evaluated together with runtime stability, output quality, and end-to-end speed rather than assumed from precision alone.

Notes

• The repo supports multiple backends, but backend and model coverage are model-dependent.
• GGUF is a container, not a universal architecture adapter. Existing llama.cpp or whisper.cpp GGUF files are not automatically compatible unless their tensor names and model metadata are mapped to the audio.cpp family implementation.
• Build_xcframework.sh is outdated; Metal and Apple XCFramework packaging still need to be retested after the framework refactor.