Star 历史趋势
数据来源: GitHub API · 生成自 Stargazers.cn
README.md

Basalt

Basalt is a local inference engine for large language models: Qwen3.8-Flash-Next, DeepSeek-V4.1-Flash, Qwen3.8-27B, Hy4-flash and Inkling 2 Mini. It runs on an NVIDIA Blackwell GPU (sm_120) on Linux and Windows x64, with what does not fit in VRAM (the experts of a mixture-of-experts model) in system RAM, and optionally on more GPUs. The server speaks the OpenAI and Anthropic APIs. Website: basalt.jgulfo.com.

The Basalt web UI while a reply streams

Help wanted: spare Blackwell compute for testing? See Contributing.

Sponsor on GitHub Support on Ko-fi Donate on PayPal

Requirements

  • Linux x86-64 with AVX2, or Windows 10 1903+ x64 (platforms).
  • An NVIDIA Blackwell GPU (compute capability 12.0) with CUDA 13.
  • CMake 3.24+, GCC 13+ (Linux), Python 3.10+, and enough system RAM for the experts that do not fit in VRAM.

Quick start

Prebuilt Linux and Windows archives are attached to each release (what they need); otherwise build from source:

git clone https://github.com/jesdga95/basalt && cd basalt
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j                  # bin/basalt, bin/basalt-bench, bin/basalt-probe, bin/basalt-vision
                                        # Windows: add -G Ninja to the first line, from an x64 VS 2022 prompt ([platforms](docs/platforms.md#building-on-windows))
                                        # one model only: -DBASALT_MODELS=qwen4exp, deepseek41, qwen35, hy4flash or inkling2 ([building](docs/building.md#building-only-some-models))
pip install -r requirements.txt
hf download jesdga/Qwen3.8-Flash-Next-Basalt Qwen3.8-FN-NVFP4-DQ6_K.basalt --local-dir models

Put the settings in basalt_config.json:

{
  "version": 2,
  "models": [{ "model": "models/Qwen3.8-FN-NVFP4-DQ6_K.basalt", "gpu": "0,1", "kv": "int8" }],
  "engine": { "resident-experts": true },
  "server": { "port": 8080 }
}
bin/basalt serve --config basalt_config.json

The server listens on 127.0.0.1. Every setting: configuration. More examples: example configurations.

Models

Qwen3.8-Flash-NextDeepSeek-V4.1-FlashQwen3.8-27BHy4-flashInkling 2 Mini
Packsdefault Qwen3.8-FN-NVFP4-DQ6_K, Hugging Facedefault DeepSeek-V4.1-Flash-1017-UD-IQ1_M, Hugging FaceQwen3.8-27B-QUASAR-NVFP4, Hugging FaceHy4-flash-NVFP4, Hugging FaceInkling-2-Mini-NVFP4-DQ8, Hugging Face
Contextup to 262,144 (longer with YaRN)up to 1,048,576up to 262,144up to 1,048,576up to 1,048,576
DraftingMTP, on by defaultDSpark, on by defaultDFlash 2, on by defaultDFly2, on by defaultDSpark, on by default
Imagesyesyesyesyesyes
Documents (PDF, text)yesyesyesyesyes
Guideguideguideguideguideguide
Precisionprecisionprecisionprecisionprecisionprecision

All models compared, with each one's context limit, flags and cards: models.

Video

Qwen3.8-Flash-Next and Qwen3.8-27B read video; DeepSeek, Hy4-flash and Inkling 2 Mini do not. A video is named like an image in a chat message, and needs --vision and FFmpeg on the machine. Flash-Next decodes frames on the GPU with NVDEC; the 27B decodes them on the CPU. How a request names a video, and the limits: video.

Documentation

Credits

Thank you to everyone who has contributed or sponsored the project.

License

MIT, see LICENSE.

关于 About

Local inference engine optimized for NVIDIA Blackwell

语言 Languages

C++62.1%
Cuda18.7%
Python16.4%
CMake1.0%
C0.9%
JavaScript0.4%
CSS0.2%
Jinja0.1%
Shell0.1%
HTML0.0%
Dockerfile0.0%

提交活跃度 Commit Activity

代码提交热力图
过去 52 周的开发活跃度
3328
Total Commits
峰值: 794次/周
Less
More

核心贡献者 Contributors