TurboServe

TurboServe: Serving Streaming Video Generation Efficiently and Economically

This repository provides a trace-driven demo of TurboServe's core scheduling and runtime workflow.

Introduction

Streaming video generation serves long-lived, interactive sessions that emit video progressively, chunk by chunk. Unlike offline video generation or conventional LLM serving, each session must preserve generation state across active and idle periods while satisfying a tight per-chunk latency target. In multi-user GPU clusters, heterogeneous session durations gradually create load imbalance, while bursty user activity makes static GPU provisioning either expensive or unable to sustain real-time generation.

TurboServe is a serving system designed specifically for this workload. It formulates streaming video serving as an online scheduling problem that jointly coordinates session placement and GPU provisioning. Its closed-loop scheduler combines migration-aware placement to reduce bottleneck per-chunk latency with load-driven autoscaling to adapt GPU capacity as demand changes.

At runtime, TurboServe supports coalesced chunk processing, GPU-CPU offloading for idle sessions, and NCCL-based GPU-GPU state migration. Evaluations on production traces from Shengshu Technology, multiple model sizes, and clusters of up to 64 NVIDIA B300 GPUs show average reductions of 37.5% in worst-case per-chunk latency and 37.2% in total GPU operating cost over baseline serving configurations.

Installation

Use Python 3.10+ and install a CUDA-enabled PyTorch build before the remaining dependencies. The provided runtime configuration requires eight visible NVIDIA GPUs.

pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

Usage

Run TurboServe with autoscaling and migration enabled:

python -m turboserve.demo \
  --config_path configs/longlive_multiplexing_worker_gpu_ablation_random_f1.yaml \
  --runtime_config configs/runtime_8gpu_process_nccl.yaml \
  --trace_in traces/example_8gpu.json \
  --random_weights 1 \
  --enable_autoscaling 1 \
  --enable_migration 1 \
  --max_wall_s 300 \
  --summary_path summary_turboserve.json

When both autoscaling and migration are disabled, TurboServe runs the fixed full-pool least-load admission baseline:

python -m turboserve.demo \
  --config_path configs/longlive_multiplexing_worker_gpu_ablation_random_f1.yaml \
  --runtime_config configs/runtime_8gpu_process_nccl.yaml \
  --trace_in traces/example_8gpu.json \
  --random_weights 1 \
  --enable_autoscaling 0 \
  --enable_migration 0 \
  --max_wall_s 300 \
  --summary_path summary_baseline.json

Configuration

Important files:

• configs/runtime_8gpu_process_nccl.yaml: replay/runtime constants, scheduler knobs, latency model parameters, coordinator timeouts, and process-NCCL settings.
• configs/longlive_multiplexing_worker_gpu_ablation_random_f1.yaml: 1.3B random-init model configuration used by the examples.
• traces/example_8gpu.json: example 8-GPU workload trace.
• prompts/workload_4.jsonl: prompt pool reused by replayed sessions.

Key controls:

• --random_weights: skip checkpoints and use random generator/T5/VAE initialization.
• --enable_autoscaling: enable dynamic GPU budget sizing.
• --enable_migration: enable physical session-state migration.
• scheduler.turboserve_capacity_per_gpu: placement capacity used by the scheduler.
• runtime.capacity_per_gpu: capacity used by runtime accounting and pass-rate normalization.

Citation

If you find our work valuable, please cite:

@article{jiang2026turboserve,
  title={TurboServe: Serving Streaming Video Generation Efficiently and Economically},
  author={Jiang, Youhe and Wang, Haoxu and Bao, Haotong and Jiang, Kai and Chen, Jianfei and Zhu, Jun and Fu, Fangcheng and Zhang, Jintao},
  journal={arXiv preprint arXiv:2606.19271},
  year={2026}
}

@article{zhang2026vidu,
  title={Vidu S1: A Real-Time Interactive Video Generation Model},
  author={Zhang, Jintao and Jiang, Kai and Chen, Jintao and Wang, Xu and Luo, Yang and Wang, Yuji and Chen, Dechuang and Li, Jungang and Ye, Chengyang and Chen, Marco and others},
  journal={arXiv preprint arXiv:2607.03118},
  year={2026}
}

@article{zhang2025turbodiffusion,
  title={TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times},
  author={Zhang, Jintao and Zheng, Kaiwen and Jiang, Kai and Wang, Haoxu and Stoica, Ion and Gonzalez, Joseph E and Chen, Jianfei and Zhu, Jun},
  journal={arXiv preprint arXiv:2512.16093},
  year={2025}
}