Blog | Documentation | Quick Start | Cookbook | SGLang | Join Slack
⭐ Star SGLang-Omni to help more builders discover open infrastructure for multimodal and speech serving!
News
- [2026/06] 🔥 MOSS-TTS Local Transformer v1.5 runs on SGLang-Omni with native-streaming 48 kHz speech. [Blog] [Cookbook]
- [2026/06] 🔥 Higgs Audio v3 TTS runs on SGLang-Omni for real-time, controllable speech for voice agents. [Blog] [Cookbook]
About
SGLang-Omni is a multi-stage serving runtime for omni, speech, and TTS models. Its design target is multi-stage decoding: generation split across heterogeneous stages with different compute patterns, dependency structures, and resource needs. SGLang-Omni owns the pipeline topology, stage lifecycle, inter-stage transport, model-family integration layer, and OpenAI-compatible serving surface, while composing with SGLang for high-performance autoregressive scheduling and model execution where applicable.
- Multi-stage runtime: SGLang-Omni models generation as coordinated stages: preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators.
- Stage-specialized scheduling: Each stage runs behind a scheduler matched to its workload, from SGLang-backed autoregressive scheduling to lightweight preprocessing and streaming vocoder loops.
- Transport-aware execution: A control plane coordinates requests while the relay data plane moves tensor payloads across shared-memory, NCCL, NIXL, and Mooncake backends.
- API surface: OpenAI-compatible endpoints expose multimodal chat, speech generation, batch speech, streaming speech, uploaded voices, and transcription.
What SGLang-Omni Serves
- Omni chat and speech: Run models such as Qwen3-Omni and Ming-Omni with multimodal inputs, text/audio outputs, and thinker-talker generation pipelines.
- Speech generation: Serve Higgs Audio v3, MOSS-TTS, MOSS-TTS Local, Fish Speech S2-Pro, Qwen3-TTS, Voxtral TTS, and related TTS systems through speech, batch speech, streaming speech, and uploaded-voice APIs.
- Audio transcription: Serve Qwen3-ASR through an OpenAI-compatible transcription path with documented serving and benchmarking flows.
- SGLang-Omni Router: Serve multiple Omni servers behind one OpenAI-compatible endpoint, with health checks, readiness tracking, worker lifecycle control, and model-capability discovery across the worker pool. See the Router guide.
Additional model guides, including experimental and research-oriented paths, are available in the Cookbook.
Quick Start
Community & Support
SGLang-Omni welcomes contributors working on inference systems, kernels, scheduling, inter-stage communication, model runners and cache efficiency, model integration, benchmarking, production deployment. Join the SGLang Slack or read the developer reference.
Organizations interested in supporting SGLang-Omni, TTS, or omni model serving can contact Chenyang Zhao at zhaochenyang@lmsys.org.
Acknowledgments
SGLang-Omni builds on the SGLang ecosystem and on open model work from the TTS, speech, and omni-model communities. We thank the model teams, systems contributors, and partner organizations helping make open multimodal serving faster, more reliable, and easier to extend.