NInfer

Up to 219 decode tok/s from Qwen 3.8 27B on a single V100. With software NVFP4 on Volta.

NInfer is a from-scratch C++/CUDA inference engine optimized for selected Qwen checkpoints on NVIDIA Tesla V100.

It supports text, image, and video input through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs. The runtime is intentionally narrow: one GPU, one resident model, 1–8 active requests.

Models

Artifacts contain the exact model weights, tokenizer, chat template, and required media frontend resources.

Performance

Qwen3.8-27B NVFP4 reaches 218.98 decode tok/s at K=1, with 99.2% MTP draft acceptance. That result is on a V100-PCIe-32GB, not SXM. The equivalent SXM2 card is roughly 7% faster; decode is predominantly HBM-bound, so PCIe bandwidth and host performance have little effect.

Tesla V100: software NVFP4 and groupwise inference

The Qwen3.8-27B NVFP4 short-context target round, retested after the width-6+ verify fix below, peaks at K=1 draft tokens: 219.0 decode tok/s at 99.2% draft acceptance -- narrow windows win outright on this corpus; see the full K sweep below.

The single-request sweep uses the public Engine benchmark on a Tesla V100-PCIe-32GB with CUDA 12.8 and INT8 group-64 KV. Prefill is an isolated pp2048 run; decode is pp2048+tg256 with CUDA Graphs and the optimized proposal head. Each result uses one discarded warmup and three measured repetitions. On the DFlash window sweep in the V100 notes the preferred V100-SXM2-32GB ran about 7% faster per round; this decode workload is HBM-bound, so the host and PCIe bus barely matter.

Full Qwen3.8-27B nvfp4 MTP draft-window sweep on this same corpus-continuation shape, now that the width-6+ fix removes the sm_70 cap at four:

This corpus is a deterministic continuation with unusually high, near-ceiling acceptance at every K, so narrow windows win outright: round-verify cost dominates once there's little more accepted length to buy. Treat these as a synthetic-corpus ceiling, not a general-generation rate -- on real, less predictable text the practical production sweet spot is K=3 (see the long-context sweep elsewhere in this repo's history).

Decode throughput depends strongly on draft acceptance -- see the MTP sweep above for how much. The sm_70 width-6+ target-verify regression is fixed (see below), so --spec mtp now accepts the same [1,7] window upstream does, no Volta-specific cap. --spec dflash2 peaks at K=7 on this same corpus-continuation shape (a 3-10 sweep falls off on both sides); MTP still leads DFlash2 here at every K tried.

The Qwen3.8-27B artifacts also bundle DFlash2, the upstream masked-block speculative decoder (--spec dflash2). MTP remains the recommended Volta backend for general decoding. On a varied, non-repetitive corpus, DFlash2 K=7 narrowly beat the best MTP window at 2K and 32K context, while MTP led at 8K, 16K, and 150K. DFlash2 K=7 is its strongest static default; acceptance-driven window adaptation can favor K=3 on difficult continuations. See the full comparison in the V100 port notes.

MTP automatically extends verification up to fifteen draft tokens when the generated suffix exactly matches an earlier 16-token span and the learned proposal agrees with the lookup continuation. On a 172-token verbatim-copy prompt, Qwen3.8-27B NVFP4 produced the exact continuation at 201.0 tok/s, averaging 12.91 output tokens per round. This is a context-reproduction fast path; ordinary generation continues to use the normal MTP window and the general decode results above.

Context-lookup MTP was inspired by syv-ai/qwen38-27b-rtx3090.

On Volta, NVFP4 is decoded and executed in software using tuned FP16 tensor-core and SIMT kernels. Dense MLP gate/up payloads are prepacked in place during model load for the QPN decode layout; the artifact on disk is unchanged and inference does not perform runtime weight repacking. The Volta QPN prepacking work was inspired by dnv2003/v100-skinny.

The 35B-A3B production DFlash round at a 2,048-token context uses K=3: 125.9 tok/s and 3.8 mean output tokens per round over ten measured rounds after two warmups.

Quick start

NInfer requires 64-bit Linux, a Tesla V100 with CUDA Toolkit 12.8, CMake 3.28 or newer, a C++20 host compiler, Ninja, pkg-config, FFmpeg development libraries (libavformat >= 60, libavcodec >= 60, libavutil >= 58, and libswscale >= 7), and libcurl >= 7.85. This port builds for sm_70.

Select CUDA 12.8 and Volta explicitly:

cmake -S . -B build-v100 -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc \
  -DCMAKE_CUDA_ARCHITECTURES=70
cmake --build build-v100 -j

The same five .ninfer artifacts use the public Engine/CLI/serving routes. See the V100 port notes for qualification and the preferred-GPU launcher.

On Volta, loading an NVFP4 artifact repacks each dense MLP gate/up and down payload in place on the GPU into the QPN fragment order used by the decode kernels. Row-scaled FP8 payloads are likewise permuted into their native Volta QPN stream. The .ninfer file and host-side artifact bytes are unchanged, the device allocation remains the same size, and temporary repack storage is released before inference. These device-resident weights are therefore deliberately mutated at load; the V100 path does not retain checkpoint-native immutable layout for those payloads.

Tests, benchmarks, and maintainer tools are excluded from the default build. There is no install target or packaged binary distribution; run NInfer from its source build tree.

Download the artifact used by this example with the Hugging Face CLI:

hf download neroued/Qwen3.8-27B-nvfp4-NInfer \
  qwen3_8_27b_nvfp4.ninfer \
  --local-dir models

Start a long-running text/vision agent server with one active-request lane and Device/Host checkpoint retention:

./build/apps/ninfer-serve models/qwen3_8_27b_nvfp4.ninfer \
  --max-context 240000 \
  --prefill-chunk 2048 \
  --kv-capacity auto \
  --max-concurrency 1 \
  --kv-dtype int8 \
  --device-state-slots 1 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 4 \
  --lm-head-draft \
  --preserve-thinking \
  --vision

Each request has a 240,000-token logical ceiling. The Device KV pool is sized from the memory left after weights, workspace, state, Vision, and graph allocations. The cache tiers provide one Device checkpoint slot, eight pinned Host State slots, and 8 GiB of pinned Host KV.

Send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.8-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

Run a one-shot CLI request with a 32,768-token allocation:

./build/apps/ninfer models/qwen3_8_27b_nvfp4.ninfer \
  --prompt "Explain prefill and decode, then give a concise conclusion." \
  --max-context 32768 \
  --max-new 8192 \
  --kv-dtype fp8 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Answer content is written to stdout. Structured startup/runtime-error records and the CLI-owned reasoning, timing, throughput, memory, and speculative-decoding report are written to stderr; reasoning and the result report remain unprefixed product output. On a terminal, weight materialization additionally uses one transient progress line. Redirected stderr receives only persistent structured phase records, including rate-limited progress for long loads. Option and local input errors remain direct command diagnostics. Use --messages FILE and --vision for structured image/video input; see the CLI guide and committed examples.

Resource-aware long-context reuse

A reusable prefix checkpoint contains KV and the complete continuation state for its exact prompt frontier. A Device-resident checkpoint resumes directly. Under pressure, the planner weighs Device retention, pinned Host State/KV, and eviction by immediate restore work and later reuse cost. Active requests retain their completion reservations.

See Resource scheduling and context cache for the algorithm and Serve TTFT benchmark for public-HTTP coverage of hot reuse, Host resume, eviction, shared prefixes, scheduling boundaries, and multimodal load.

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

The Qwen3.6 rows used temperature 0.6 and presence penalty 1.0; the Qwen3.8 rows used temperature 1.0 and presence penalty 0.0. Multimodal evaluation used --vision and an 81,920-token context limit. Text evaluation used 262,144 tokens except Qwen3.8-27B NVFP4, which used 252,928 tokens for its measured memory envelope. Each score is one sample per problem; model cards contain the correct/total counts and evaluation notes.

Startup notes

GPU residency is fixed at process startup. --spec selects speculative decoding residency, and --vision selects Vision residency. DFlash is available for text-only Qwen3.6-35B-A3B execution, and DFlash2 for Qwen3.8-27B.

Docker

Build the runtime image on a host with the NVIDIA Container Toolkit:

docker build --tag ninfer:local .

Mount the downloaded model and run the same example server profile:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_8_27b_nvfp4.ninfer \
  --host 0.0.0.0 \
  --max-context 240000 \
  --kv-capacity auto \
  --max-concurrency 1 \
  --kv-dtype int8 \
  --device-state-slots 1 \
  --host-state-slots 8 \
  --host-kv-mib 8192 \
  --spec mtp --draft-tokens 4 \
  --lm-head-draft \
  --preserve-thinking \
  --vision

Capabilities and limits

All registered model IDs support:

• text generation with thinking and non-thinking prompt modes;
• image, multi-image, video, and mixed multimodal messages;
• chunked prefill, exact-batch CUDA Graph decode, and startup-bounded batched decode;
• MTP speculative decoding with draft windows from one to seven;
• DFlash2 masked-block speculative decoding for Qwen3.8-27B (from the upstream integration), draft windows from one to fifteen;
• BF16, INT8, and FP8 KV storage;
• offline causal-perplexity scoring;
• private and shared exact-prefix reuse with Device/Host State and KV retention;
• model-aware sampling defaults and explicit sampler overrides;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming, tools, local response state, token counting, and usage accounting.

The 35B-A3B target additionally supports text-only DFlash with draft windows from one to fifteen.

The product boundary remains intentionally small:

• one Tesla V100 and one resident model per Engine;
• a startup-fixed capacity of one to eight active requests with bounded FIFO ingress;
• no request preemption, priority/QoS, active-request swapping, weight offload, multi-GPU, or distributed serving;
• one shared startup-fixed KV pool across active requests and retained prefixes;
• no runtime model discovery or unregistered checkpoint fallback;
• parsed tool calls are returned to the client; NInfer does not execute tools;
• the in-tree C++ headers are not distributed as an installed SDK.

--max-context is each sequence's logical limit. --kv-capacity sizes the shared Main Text KV pool used by active requests and retained prefixes; auto resolves the largest legal capacity at startup from the memory remaining after weights while keeping 1 GiB of sizing headroom. Explicit capacities remain fixed for the process lifetime.

Documentation

• Documentation index
• CLI
• HTTP serving
• V100 qualification and performance
• Perplexity evaluation
• Resource scheduling and context cache
• Serve TTFT benchmark
• CLI examples
• Contributing

Run the relevant --help for the exact current option contract.

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The Qwen3.6-27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. The Qwen3.8-27B NVFP4 artifact also uses the fixed mixed FP8/NVFP4 weights from unsloth/Qwen3.8-27B-NVFP4. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.