Star 历史趋势
数据来源: GitHub API · 生成自 Stargazers.cn
README.md

AIBrix

License Documentation Release Slack

Documentation · Blog · White paper · X · Slack

AIBrix is a Kubernetes control plane for production LLM inference. You bring the engine — vLLM, SGLang, or TensorRT-LLM — and AIBrix routes each request from live engine state, scales on inference metrics, and manages models, LoRA adapters, and KV cache.

Use it when you run many models or replicas, mix GPU types, or need to hold cost and latency targets.

Features

CapabilityWhat it does
Gateway and routingOpenAI-compatible API. Picks a pod from live engine state, including prefix-cache, load, and prefill/decode placement, with session affinity for agentic workloads.
Multi-engineServe vLLM, SGLang, and TensorRT-LLM behind one gateway.
LoRAPack many low-rank adapters onto shared base-model pods.
Model lifecycleDeclare models with ModelClaim, and preload images and weights with ModelWarmup so scale-up is faster.
AutoscalingScale replicas on engine metrics such as queue depth, KV cache, and tokens.
Distributed inferenceRun multi-node and multi-role deployments, including prefill/decode, with StormService.
Distributed KV cacheOffload KV cache and reuse it across engines.
Heterogeneous GPUsMix GPU types and still target cost and SLO. Experimental.
RuntimeSidecar that downloads weights, loads and unloads adapters, and normalizes engine metrics.

Architecture

AIBrix architecture: a data plane that routes requests to engine pods, and a Kubernetes control plane for model lifecycle, warmup, multi-node serving, autoscaling, and observability

Requests hit an OpenAI-compatible gateway and go straight to an engine pod. Kubernetes controllers decide what should be running: model lifecycle, warmup, multi-node topology, autoscaling, and observability. The architecture guide walks through both planes.

Quick start

These commands install AIBrix, including its Envoy Gateway dependency, into the cluster your current kubectl context points at. KubeRay is included and is only required for Ray-based multi-node inference.

The snippet installs the latest release, v0.8.0. Change VERSION to install a different release.

Install a release

VERSION=v0.8.0

# Dependencies (Envoy Gateway, KubeRay, and friends)
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-dependency-${VERSION}.yaml" --server-side

# CRDs ship separately so uninstalling the operator does not delete user resources
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-core-crds-${VERSION}.yaml" --server-side

# Controllers, gateway plugins, and supporting services
kubectl apply -f "https://github.com/vllm-project/aibrix/releases/download/${VERSION}/aibrix-core-${VERSION}.yaml"

kubectl get pods -n aibrix-system

When the pods are ready, deploy a model and call the gateway. The quickstart has a base-model manifest and the OpenAI-compatible curl commands. Helm install steps are in the installation guide.

Install from this checkout

Images in config/default are tagged nightly.

git clone https://github.com/vllm-project/aibrix.git
cd aibrix

kubectl apply -k config/dependency --server-side
kubectl apply -k config/crd --server-side
kubectl apply -k config/default

Documentation

Full guides live at aibrix.readthedocs.io.

You want toStart here
Serve a first modelOverview, then the quickstart
Make a running deployment cheaperAutoscaling, LoRA, KV cache, observability
Hit scale or latency targetsPrefill/decode, multi-node inference, heterogeneous GPUs

News

Releases

Older releases

Talks

Community

Contributions are welcome. Read the contributing guide, talk in #aibrix on the vLLM Slack, or email maintainers@aibrix.ai.

Bugs and feature requests go on GitHub Issues. AIBrix is licensed under Apache 2.0.

关于 About

Cost-efficient and pluggable Infrastructure components for GenAI inference

语言 Languages

Go56.7%
Python29.6%
Jupyter Notebook6.5%
TypeScript5.3%
Cuda0.4%
Shell0.4%
CSS0.4%
Makefile0.2%
HCL0.1%
JavaScript0.1%
CMake0.1%
HTML0.0%
Dockerfile0.0%
C++0.0%
Go Template0.0%
C0.0%

提交活跃度 Commit Activity

代码提交热力图
过去 52 周的开发活跃度
1033
Total Commits
峰值: 99次/周
Less
More

核心贡献者 Contributors