Qwen3.8-Flash-Next on 2× RTX 5090 (ExLlamaV3 + TabbyAPI)

Serving configuration, launcher, image recipe, kernel overlays, instruments and measurements for Qwen3.8-Flash-Next on two RTX 5090 cards, served as qwen3.8-flash-next-exl3-2.68bpw-selfcal.

The serving configuration supports concurrent coding-agent sessions with growing contexts. The launcher sets eight slots and a shared KV page pool; the measurements below cover decode, cold prefill, standard serving traffic and quality checks on that configuration.

• Checkpoint: turboderp's self-calibrated 2.68 bpw EXL3 quant, revision 2.68bpw_h6_ng8, 6-bit output head, 8-bit n-gram table and 2.5-bit MTP.
• Engine and server: ExLlamaV3 dev 5783a93 (v1.5.2) plus this repository's overlays, under TabbyAPI 53da7919; image tabbyapi:tune268-r2, 50 launcher selectors (docker/).
• Capacity: 262,144-token request window, 819,200-token shared page pool, 8-bit KV, eight slots and layer split [30, 30].
• Drafting: the checkpoint's MTP head, depth three through four jobs and depth two at five to eight; prompt lookup enabled at actual batch one after probation.
• Interface: vision, separate reasoning and qwen3_coder tool parsing configured by scripts/launch-flashnext.sh.

Measurements are dated 2026-10-23 and linked to their published records in the served 2.68 write-up. The experiment index is bench/RESULTS.md.

Numbers

Measured 2026-10-23 on tabbyapi:tune268-r2, the 2.68 bpw self-calibrated checkpoint, pool 819,200 and all 50 served selectors: run 20261023T000904Z supplies decode, prefill and depth; run 20261023T005148Z supplies standard serving. fn_bench records greedy code/prose bursts of 1,024 forced tokens at one to eight streams; vllm bench serve records closed-loop ShareGPT and Spec-Bench traffic. Decode rate excludes time to the first token; decode aggregate is the mean round sum of concurrent request rates, and standard aggregate is output tokens over wall time (method).

Solid lines show decode alone on code and prose; dashed lines show ShareGPT and Spec-Bench with new requests' prefill interleaved. The decode-only figure uses the same decode records.

• At one stream, code / prose decode is 248.7 / 251.4 t/s. At eight streams, code / prose decode aggregate is 795 / 786 t/s, with median per-stream rates 98.8 / 98.2 t/s; each request forces 1,024 greedy tokens on a 118 / 106-token prompt (20261023T000904Z).
• At eight streams, ShareGPT / Spec-Bench output is 445.9 / 492.7 tok/s over wall time, with per-stream 1000 / TPOT p50 of 66 / 69 t/s; output lengths follow the ShareGPT references or force 256 tokens for Spec-Bench (20261023T005148Z).
• Decode-burst TTFT medians are 0.163 / 0.155 s at one stream and 0.919 / 0.910 s at eight (code / prose, greedy, 1,024 forced tokens; 20261023T000904Z).

Cold prefill averages 12,440 prompt tokens/s at 199,977 prompt tokens (16.076 s TTFT), and 12,356 prompt tokens/s at 239,866 (19.413 s); each point is the mean of three salted prompts with 64 forced greedy output tokens (20261023T000904Z, unit b). Decode at depth averages each cold/cached pair at one stream with 2,048 forced greedy tokens (unit c).

Conditions

The decode curve has two boots, one unrecorded warm-up and three measured rounds per shape, 432 requests, distinct stream suffixes and 118 / 106-token code / prose prompts. Prefill has three salted prompts per target through 240k; depth has two requests per shape at one stream. The standard matrix has two replication passes, a fresh boot per cell and 7,040 measured requests at one, two, four and eight concurrent requests. All units use the served boot warm-up, NVMe off, stock 600 / 575 W power limits, core offset zero and memory offset +4500 MHz, with independent per-boot copies of the existing tune cache (method and boot readbacks).

Served configuration

• Image tabbyapi:tune268-r2, pinned to sha256:2239954c372397b97ec3d570d887fe0afe679e8de214dfda558f5095d8f855fd; 50 launcher selectors. The launcher, docs/CONFIG.md and image chain specify the configuration; history records the promotion.
• Checkpoint qwen3.8-flash-next-exl3-2.68bpw-selfcal, turboderp revision 2.68bpw_h6_ng8: experts K3 on layers 0–3/44–47, K2.5 on 4–43, dense/shared K6, 6-bit output head, 8-bit n-gram table and 2.5-bit MTP.
• 262,144-token window, 819,200-token shared pool, eight slots, 8-bit KV, layer split [30, 30], draft component on GPU 1 and draft policy [[4, 3], [8, 2]].
• The launcher enables EXL3_DENSE_V2_K6=1, EXL3_MOE_COOP_HALF=1 and EXL3_MOE_PREFILL_E3_HALF=1; the round-2 histogram, static and uncapped alternatives remain at their image default 0. All six selectors default off in their overlay recipes.
• Prefill merge, asynchronous recurrent stashes, tail checkpoints, resumable whole-prompt prefill, adaptive prompt lookup and cache trace are enabled; NVMe tier is off. Kernel caches use /srv/qwen5090/.exl3cache-tune268; the launcher performs the served boot warm-up.

What the stack is

The checkpoint is turboderp's self-calibrated EXL3 quantization of Qwen3.8-Flash-Next. ExLlamaV3 dev 5783a93 and TabbyAPI 53da7919 supply the inference engine and OpenAI-compatible server. The complete source overlay chain is in docker/, with attribution and licences in THIRD_PARTY.md.

The engine overlays cover multi-job QSA, concurrency-indexed MTP drafting, cooperative MoE decode, dense projections, the hyper-connection mixer, two-card prefill, recurrent checkpoints and prompt lookup. The 2.68 overlays add K6 dense V2 twins, K2.5 cooperative decode and K2.5 grouped E3 prefill. The build verifies patch hashes and predecessor SASS; the launcher enables the measured selectors (recipe).

TabbyAPI carries the reasoning-loop handling, one prompt encode per request and off-loop encoding above 12,000 characters. Sampler fallbacks are temperature 1.0, top_k 20 and top_p 0.95 with force: false; client settings take precedence. Vision, reasoning and tool parsing are enabled in the generated configuration (settings).

Hardware

Read from the box on 2026-09-19. Every number in this README was measured on this hardware and driver; the memory clock offset is stated per period below.

• Host: ASRock X870 Taichi Creator, AMD Ryzen 7 9800X3D, 64 GB DDR5-6000 (2 × 32 GB), Ubuntu 24.04.4 LTS, kernel 7.0.0-30-generic.
• GPUs: two RTX 5090 32 GB (sm_120) on PCIe Gen5 x8/x8. Power limits are the cards' defaults, 600 W (ASUS, cuda:0) and 575 W (HP OEM, cuda:1). Memory clock offset +4500 MHz on both cards, core clock stock. A boot-time service applies them once per host boot, and since 2026-09-25 the launcher sets the memory offset before every engine boot and logs the readback. The offset had reset to 0 between 2026-09-03 and 2026-09-19 without a reboot, so the numbers measured from 2026-09-19 to 2026-09-25 02:22 UTC ran at the stock memory clock; +4500 is +14.3 % DRAM bandwidth and +1.7 to +1.8 % decode per stream at 1 stream (R726).
• Driver: NVIDIA 610.57.04 open kernel modules, CUDA 13.3 user-mode driver.
• Storage: one KIOXIA KBG80ZNV2T04 2 TB NVMe (ext4) holds the checkpoint, including the n-gram table that decode reads rows from; the NVMe prefix tier is disabled by default. A sequential 16 MiB O_DIRECT read of a tier segment ran at 6.7 GB/s (2026-09-19).

Measured and not served

Each entry names the change and the number that kept it out of the served configuration. c1, c4 and c6 mean 1, 4 and 6 concurrent streams.

• A windowed MTP draft cache, a sink page plus the last 16,384 tokens per slot (in the image, off): +3.4 % page pool (R579), but a prompt revived from the prompt cache in a later request group drafts 0.655 accepted per proposed token against 0.868 without it, and an agent-shaped replay decodes 2.0 % faster per stream without it (R728).
• Mixed draft depth per job inside one verify batch, so 5 to 7 streams fill the 16 verify rows: 0.79× at 5 streams and 0.84× at 7, code, 256 forced tokens, greedy. The verify window grows to the deepest job in the batch, which adds a sequential draft level to every step (R678b).
• 17 to 32 verify rows on the cooperative MoE kernels, as two calls of at most 16 rows: bit-identical, −14.5 % at 8 streams against drafting one token (R566). One launch of up to 32 rows is served since 2026-09-25 (R717).
• Draft depth 2 above 4 jobs on the 16-row decode paths: −32 to −39 % at 6 and 8 streams, because 18 and 24 verify rows fell off the cooperative MoE decode kernels (2026-09-19, R560, R562). With the rows32 paths depth 2 at 6 to 8 jobs is served since 2026-09-25 (R717).
• Draft depth 4 at 1 stream, with or without a controller: costs 32,768 page-pool tokens and returns at most about +2 % (R567).
• A deeper MTP draft for a single decoding job (depth 4 or 5 instead of 3), all arms at a 753,664-token pool: +4.8 / +5.2 % code and −5.4 / −8.2 % prose at 1 stream against depth 3, 16,384 / 49,152 fewer pool tokens than the served 819,200 of that date, and a different greedy output (2026-09-19, R537).
• Adaptive MTP draft depth, round 2: −2.4 to −3.5 % prose at 1 stream, no gain on code (R556).
• Two draft chains verified together: +5 to +6.5 % accepted tokens for twice the verify rows, modelled at −10 to −13 % (R564).
• A 4,096-token prefill chunk: does not boot beside the page pool (R574).
• Prefill chunk 1,024 or 512 instead of 2,048: running streams get 1.5× the decode frames while another request prefills, but cold prefill runs at about half the rate and the new request waits 1.4–1.7× longer for its first token (R553). Chunk 1,024 for pool size: R483, R485.
• The served stack on upstream dev 1d64111: −49,152 pool tokens and 1–3 % decode (R563). A later port onto dev 5783a93 is served since 2026-09-27, at −81,920 pool tokens (R784).
• The MTP draft's embedding copy on cuda:0: +16,384 pool tokens for −1.8 % code and −2.0 % prose at 1 stream (R555).
• The K=3 MoE decode kernel without register spills: bit-exact, slower per call in 28 of 30 kernel cells, −0.37 % code at c1 over 8 boots with a 95 % interval of −0.83 to +0.09 % (2026-09-19, R536).
• Recurrent checkpoints stored at the end of each reply: correct, but they save about 9k prefill tokens over a 120-call agent replay, below its run-to-run spread (R524).
• A fused shared-expert kernel: after the side-stream overlap the shared expert's residual is 1.9 µs per layer at 4 rows, at most 0.6 % of a 1-stream step and 1.2 % at 4 streams (R521).
• EXL3_INT8_GEMV=0: −0.3 % at c1 with a 95 % interval of ±1.2 % over 8 boots (R520b).
• 6 decode slots: +17 % at c6 and 9 % slower on an 8-agent replay, measured before the bf16 GDN state halved the per-slot cost (R518).
• Split [30, 31] at 393,216 (R487).
• The n-gram table in host RAM: +1–2 % for 30.5 GiB (R484).
• The host KV tier (R358, R493); GDN state replay (R496); a 4-bit MTP graft (R498); CPU-offloaded experts (R482); MoE coop mode 3 (R462).
• The 2026-09-18 prompt-lookup variant: +3–4 % on code at c1, flat at c4 (R501).
• K8V4: +18 % pool for −11 % code at c1 (R480).
• exllamav3#303 MTP hot vocabulary (R377); exllamav3#246 and #290 (R365); a 32-row MoE decode envelope written for this stack (R366).
• The same checkpoint on vLLM through vllm-exl3: 0.62× the c1 and 1.06–1.12× the c4 of this stack's 3.05 bpw configuration of 2026-09-18. Work on that route stopped the same day (vLLM route).

How the numbers are measured

The served publication write-up names all inputs and definitions. Decode, prefill and depth come from 20261023T000904Z, units a/b/c, on the served 2.68 configuration. fn_bench (bench/probe.py) records one line per request, forces length through min_tokens, checks finish_reason=length and retains both server completion counts and streamed frame counts.

Decode uses two boots, one unrecorded warm-up plus three recorded rounds per shape, greedy 1,024-token outputs, one to eight distinct streams, code prompts of 118 tokens and prose prompts of 106. Per-stream is the median request rate (completion_tokens - 1) / decode_window_s; aggregate is the mean round sum of concurrent request rates. The secondary round-wall aggregate includes TTFT and straggler tails. Every stream decodes together during 91.3–98.4% of the mean decode window at two to eight streams. The maximum two-boot spread is 2.56% for per-stream medians and 1.98% for decode aggregate (records).

Cold prefill uses three salted prompts at each of five targets, ignores the calibration point in the chart and measures prompt_tokens / ttft_s, with 64 forced greedy output tokens. Depth uses code filler budgets 0/74,000/148,000 and prose budgets 0/133,000/266,000; each shape has a cold/cached pair at one stream with 2,048 forced greedy output tokens. The chart uses actual prompt counts and mean decode t/s; tokens/frame is completion_tokens/client_frames, and ms/frame is 1000*decode_window_s/client_frames (prefill, depth).

Figures are drawn by bench/plot.py, with explicit run paths and missing-input errors. Run uv run bench/plot.py to regenerate decode-scaling.svg, std-bench.svg and prefill.svg.

Standard benchmark (vllm bench serve)

vLLM v0.30.0's serving benchmark runs against TabbyAPI through bench/vllm_bench_tabby.py, using ShareGPT V3 and Spec-Bench. Run 20261023T005148Z, unit d contains 16 cells, passes A/B at concurrency 1/2/4/8, 7,040 measured requests and a fresh boot per cell. Every cell uses tabbyapi:tune268-r2, all 50 selectors, pool 819,200, the served warm-up, stock power/core, memory offset +4500 and NVMe off.

Each ShareGPT cell uses the same 400 conversations at seed 7310 and forces the reference output lengths, mean 210.3 tokens; each Spec-Bench cell uses all 480 prompts across 13 categories and forces 256 output tokens. Mean input lengths are 272.255 / 321.952 tokens and maxima 1,070 / 1,540 (ShareGPT / Spec-Bench). Thinking uses the template default and sampling is greedy. All measured requests finish at length, with no failures or cached prompt tokens; each cell outputs 84,120 / 122,880 tokens.

Output tok/s is all completion tokens over the cell's wall time, including prefill and turnover. Per-stream t/s is the mean of each pass's 1000/p50_tpot_ms, where TPOT is (E2E - TTFT)/(output_tokens - 1) per request. Means pool A/B equally; p99 columns show their range. A/B spread is abs(A-B)/mean(A,B); the maximum is 0.47% at Spec-Bench c2, within the 3% publication gate. No cell is replaced by a repeat.

ShareGPT V3

Spec-Bench

τ is the server-log ratio Σ generated / (Σ generated - Σ accepted drafts), including the prefill step; it differs from the decode probe's tokens/frame and from published Spec-Bench acceptance definitions. The summary retains τ, client τ, accepted/proposed fractions, depth, ITL and category diagnostics. The detailed numeric request arrays and sample hashes are published with the scalar results; generated text and prompts are omitted.

The fixed packet validators pass on standard request counts, manifests, forced lengths, foreign-request checks and the recorded PUBLISHABLE decision. Decode, prefill and depth numeric request checks pass; their copied inputs omit the final server logs, preventing an independent repetition here of server-count and zero-cache assertions. The unit VALIDATED markers and supplied summaries are the retained upstream validation evidence (publication limits).

Reproducing a boot

bash docker/build-chain.sh              # builds through tabbyapi:tune268-r2
# Review and update R825C_IMAGE_ID in the launcher for the rebuilt image before serving.
ssh flan 'bash -s' < scripts/launch-flashnext.sh
STOP=1 bash scripts/launch-flashnext.sh

Rebuilding the image chain, running the measurements and handing the box back are in docs/RUNBOOK.md.

Repository map

Attribution and licence

The original work here (documentation, instruments, launcher, overlay installers, measurements) is MIT. The model is the Qwen team's under the Qwen Community License; the served checkpoint is turboderp's self-calibrated 2.68 bpw quant, with r0b0tlab's and other turboderp packs retained in historical measurements; the engine is ExLlamaV3 (MIT) and the server TabbyAPI (AGPL-3.0), and every kernel patch in docker/ is a derivative of one of them. The patches were written with coding agents from static source dumps and admitted or rejected by measurement on the box. Ideas were taken from vcruz305's DGX Spark recipe, DominikBucko, HaberstrohSystems and halogen; the full table is in THIRD_PARTY.md.

Links

Model and checkpoints: Qwen3.8-Flash-Next · r0b0tlab 2.50 bpw · turboderp EXL3 packs

Engine and server: ExLlamaV3 · EXL3 conversion · TabbyAPI · llguidance

Upstream pull requests measured here: exllamav3#246 · #284 · #290 · #299 · #303 · #337

Other setups of this model: vcruz305 DGX Spark recipe · vcruz305/vllm-exl3 · DominikBucko, 2× RTX 3090 · HaberstrohSystems, 24 GB SGLang

Benchmarks and harnesses: tool-eval-bench · mini-SWE-agent · SWE-bench · lm-evaluation-harness · halogen · vLLM (vllm bench serve) · ShareGPT V3 · Spec-Bench