GLM-5.3-Flash EXL3 on DGX Sparks with TensorFold

_{by Mia's AI Lab}

Serve GLM-5.3-Flash from two NVIDIA DGX Sparks (GB10, 128 GB each, linked by their ConnectX-7 ports) through an OpenAI-compatible API, with 4 concurrent requests, the model's full 1,048,576-token context and image and video input. It runs TensorFold v0.6.0 on both Sparks (one rank on each) in NVIDIA's PyTorch container, plus 77 patches (65 of v1.4, 3 for 3 Sparks, experimental, 1 for up to 8 requests at once, 1 for stopping serial requests, 5 of v1.6: a shared system prompt keeps each conversation's history, the display reservation in the pool and the pool compacting before it evicts, by ezoushen; queued requests whose client left dropped at once, by desy0305; <|assistant|> ending a reply; 1 of v1.7.1: a new agent run keeps its system prompt's state once the kept-state cap is full, diagnosed and the fix proposed by meleesciony in issue #75; and 1 fix since v1.7.1: a fresh conversation after a long one no longer runs out of memory): DFlash2 and copy drafts, 4-bit dense weights, an FP8 KV cache, faster prompt kernels, a one-shot RoCE all-gather between the Sparks, several requests over one shared cache pool, vision, tool calling, /tokenize and /metrics.

• Checkpoint: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold, Mia's AI Lab's own EXL3 quantization (routed experts at 4 bits a weight, BF16 elsewhere, ~176 GB), calibrated for how TensorFold serves it (model card)
• Drafter: incoai/GLM-5.3-Flash-DFlash2, or the checkpoint's own MTP head (DRAFTER, see Configuration)
• API model id: GLM-5.3-Flash-EXL3
• Context: 1,048,576 tokens a request; the 4 requests share an FP8 KV pool of about 2.9M tokens (2,922,496 at the measured start)
• Tool calling, structured outputs (xgrammar), /tokenize, and reasoning_effort low / high / max
• One command on the first Spark: ./start.sh sets up both Sparks and starts both ranks; ./stop.sh stops them

Performance

Two DGX Sparks at the default configuration (4 streams, 1,048,576-token window, FP8 KV cache, 4-bit dense weights, DFlash2 plus copy drafts, vision on), with the GPU clocks capped at 2,200 MHz. Decode and prefill were measured with sparkDash through the OpenAI API, from another machine on the network.

Decode (aggregate across the concurrent requests, per request, and time to first token)

Replies served 4 at a time are identical to the same requests served one at a time (11 of 11 cases staggered, and 11 of 11 sent in a burst).

Up to 8 requests at once (PARALLEL=8, v1.5; its verify window then defaults to 64 rows). Measured with sparkDash on one boot (2026-10-03); one or two requests decode as fast as at PARALLEL=4:

Eight at once: +27% prose and +32% code over four. The window matters for code (8 requests: 141.2 / 160.3 / 167.0 tok/s at 32 / 48 / 64 rows), not prose. Replies stay identical to one at a time (11 of 11 staggered and in a burst, 8 in flight; drafted == serial). The memory reserve grows with PARALLEL (see MEMORY_RESERVE_GIB), so the shared pool is ~1.5M tokens at 8 instead of 2.0-2.6M at 4, with the head's lowest free memory at 12.9 GiB under a 195k-token prompt. Two Sparks therefore stay at 4 by default; three Sparks default to 8 (3 Sparks).

Prefill

Prompt reuse (the server resumes from a kept prompt state instead of prefilling, with the same reply)

Quality (FP8 KV cache and 4-bit dense weights, see Checks)

Details on the model card.

Independent Mac Studio / MCDMA qualification

An independent community experiment extended the two-Spark setup with an M3 Ultra Mac Studio over MCDMA. Its best frozen one-request candidate reached 61.27 decode tok/s, but the 6.907 s median end-to-end time remained slower than this recipe's 6.776 s implied published C1 result. It is therefore documented as a reproducible transport and integration result, not as a performance win or a supported part of this recipe. See the qualification note and the independent recipe.

Requirements

• Two DGX Sparks (or two GB10 systems with 128 GB unified memory), with nothing else large on their GPUs: each needs ~110 GiB free memory when the server starts (start.sh warns below that; stop other GPU work).
• A direct ConnectX-7 link: a QSFP cable between the CX7 ports and an IPv4 address on each end in one private subnet (e.g. 192.0.2.1/24 and 192.0.2.2/24; ping must work), with a RoCE v2 GID (start.sh checks). One QSFP port of a Spark reaches the GB10 over two PCIe Gen5 x4 links, so it appears as two netdevs and two RoCE devices (enp1s0f0np0 / enP2p1s0f0np0, rocep1s0f0 / roceP2p1s0f0). Address both twins on each Spark, in the link's subnet or each in its own (NVIDIA's two-Spark playbook gives them different ones): both are then used, as is a second cabled port in the link's subnet. A prompt chunk's all-gather is ~1.8x faster on two rails; one rail is one x4 (~112 Gb/s of the port's 200).
• Key-based ssh from the first Spark (the head, which runs ./start.sh and the API) to the second (the worker): ssh-copy-id user@<worker> (after ssh-keygen -t ed25519 if you have no key); check with ssh -o BatchMode=yes user@<worker> true.
• Docker with the NVIDIA container runtime, your user in the docker group, and rsync, on both Sparks.
• Disk, on each Spark: ~205 GB: ~176 GB (164 GiB) for the checkpoint and ~2.3 GB for DFlash2 under ~/.cache/huggingface, ~25 GB for the image under Docker's root. prepare.sh asks for what the download still needs (files already in the cache, from an earlier revision say, do not count; 180 GB, MIN_FREE_GB, when it cannot ask the Hub what is missing) and 35 GB under Docker's root (IMAGE_FREE_GB; the sum when they share a filesystem), and checks the worker for what the copy must send. With WORKER_WEIGHTS=nfs the worker needs only the image (Worker weights over NFS).
• Optional: the hf CLI on the head (faster download) and a Hugging Face token (~/.cache/huggingface/token or HF_TOKEN). HF_TOKEN is required for the gated Ablit weights (ABLIT=1, Ablit weights).

Quick start

On the head:

git clone https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold.git
cd GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold
cp scripts/local.sh.example scripts/local.sh     # set WORKER=user@192.0.2.2 in it (and FABRIC_PEER, see below)
./start.sh

WORKER is the worker's ssh target. The ranks talk over the route to it: if it is on another network than the link, set FABRIC_PEER to the worker's CX7 address. The worker needs no copy of this repository.

The first run sets up both Sparks (see below): the image (~25 GB) on each, the checkpoint (~176 GB) and DFlash2 downloaded on the head and copied to the worker, then the CUDA kernels compile once per image (into ~/.cache/tensorfold-glm53/<image hash>). Later starts take 2 to 6 minutes to load ~80 GiB of weights on each Spark. start.sh shows each step and the server's log, runs a smoke test through both ranks, and prints GLM-5.3-Flash-EXL3 is now LIVE! on port 8888 with the endpoint.

Any OpenAI client works with base_url = "http://<head-address>:8888/v1" and the model GLM-5.3-Flash-EXL3. The model thinks before it answers (reasoning_content), so give replies enough max_tokens.

curl -s http://<head-address>:8888/v1/models
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash-EXL3",
  "messages": [{"role": "user", "content": "Write a Python fibonacci function."}],
  "max_tokens": 2000
}'

./start.sh restart                                       # restart both ranks, e.g. after changing a setting
./stop.sh                                                # stop both ranks and free their GPU memory
docker logs -f glm53-flash-tf                            # rank 0's log (here)
ssh <worker> docker logs -f glm53-flash-tf               # rank 1's log (why it exited, if it did)
curl -s http://<head-address>:8888/health                # busy flag, streams, free pool tokens

Logs of earlier runs. docker rm deletes a container's log, so stop.sh (and start.sh, before it removes a stopped container left from an earlier run, a crashed one say) first saves each rank's log, stdout and stderr with timestamps, gzipped, and prints where: ~/.cache/tensorfold-glm53/logs/<date>-<time>-rank0.log.gz on the head (LOG_DIR) and ~/.cache/tensorfold-glm53/logs/<date>-<time>-rank1.log.gz on the worker. The newest 10 of each rank are kept (LOG_KEEP; 0 saves none). Read one with zcat or zless; attach both when you report a crash.

If start.sh stops at a check:

Images and video

GLM's own vision tower runs on the head (rank 0: 1.05 GiB of bf16 weights and 0.75 GiB of workspace). Send images and videos as OpenAI-style content parts in a user message:

IMG=$(base64 -w0 photo.jpg)
curl -s http://<head-address>:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "GLM-5.3-Flash-EXL3",
  "messages": [{"role": "user", "content": [
    {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMG"'"}},
    {"type": "text", "text": "What is in this picture?"}]}],
  "max_tokens": 2000
}'

A video is a video_url part ({"type": "video_url", "video_url": {"url": "data:video/mp4;base64,..."}}). Pictures and clips can also be parts of a tool result ("role": "tool", as agents return screenshots): the model reads them inside that result (with VISION=0, it is told it cannot see them, in the template's own words). A conversation with pictures in its history resumes from its kept prompt states like a text one, as long as the pictures are the same.

A request body can be up to 96 MiB, so data URLs carry about 70 MB of pictures and clips in all. By default only data URLs are accepted; VISION_URLS=1 also lets the server fetch public https:// URLs. VISION=0 serves text only and leaves the tower's ~1.8 GiB on rank 0 to the cache.

What start.sh and scripts/prepare.sh do

./start.sh works in five steps, each shown as it runs:

1. Setup: scripts/prepare.sh, when the setup is not ready on both Sparks (first run, new patches, another model, drafter, revision, image or worker).
2. Checks: the arguments (TensorFold's own parser), the link, the previous server (stopped on restart, or when only one rank is up; a stopped container left from an earlier run is removed, its log saved first), the port, free memory (a warning below ~110 GiB on either Spark).
3. Launch: rank 1 on the worker over ssh, then rank 0 and the API here, with the settings from scripts/config.sh.
4. Loading: rank 0's log as it comes, and every 15 s the elapsed time and how much of the startup estimate is on each Spark's GPU; if a rank stops, both ranks' last log lines and why.
5. Smoke test: one chat completion through both ranks, then the LIVE message and the endpoint.

A running server is left alone; ./start.sh restart stops it only after the setup and argument check pass, so a typo leaves it running (running requests are cut off; stop.sh warns). Extra arguments go to tensorfold serve on both ranks after the defaults, so they win (./start.sh restart --max-tokens 16384); ./start.sh --help lists all. FOREGROUND=1 ./start.sh stays attached to rank 0's log and exits with its exit code (for a systemd unit), without the progress lines, the window retry or the smoke test; when either rank ends, it stops the other one.

scripts/prepare.sh does the one-time setup, and is safe to re-run (each step skips work already done):

1. Preflight on both Sparks: Docker, the GPU, rsync, key-based ssh, the RoCE link, disk space.
2. The image tensorfold-glm53:v0.6.0 on the head: TensorFold v0.6.0 with every patches/*.patch applied, plus PyAV (video decoding) and xgrammar (structured outputs), on NVIDIA's nvcr.io/nvidia/pytorch:26.07-py3. It first pulls the published image ghcr.io/miaai-lab/glm-5.3-flash-exl3-2x-dgx-sparks-tensorfold:v0.6.0-<image hash>, by the digest pinned in scripts/config.sh (IMAGE_TAG / IMAGE_DIGEST) while the patches are this release's (the hash covers the patches and those pip packages); after you change patches/, it pulls that hash's tag if one is published, else (or with PULL=0) it builds.
3. The same image on the worker: pulled, else streamed from the head (docker save | docker load), checked identical.
4. The checkpoint, and DFlash2 with DRAFTER=dflash2, downloaded into ~/.cache/huggingface on the head at their pinned revisions (the checkpoint checked with tensorfold info), then copied to the worker with rsync over ssh and checked file by file. Both resume.

scripts/prepare.sh             # set up both Sparks without starting the server
scripts/prepare.sh --rebuild   # rebuild the image from scratch
PREPARE=1 ./start.sh restart   # force prepare.sh, then restart; PREPARE=0 skips the check

After changing patches/, scripts/publish-image.sh pushes the new image to GitHub Container Registry (latest and v0.6.0-<image hash>, the tag prepare.sh looks for).

KV pool and memory

GLM-5.3-Flash keeps a compressed latent cache (DSA) and the sparse indexer's pooled keys for every token, plus small recurrent states for its linear-attention (KDA) layers. With PARALLEL above 1, all requests draw their per-token caches from one shared pool:

Any one request can grow to the full window, and the four together share the pool: e.g. one 1M-token conversation next to one more of 1M, or next to three of ~640k. A request the pool cannot place yet waits until others finish (kept prompt states give way first, least recently used first, and only while the free rows fall short: free rows in several ranges are gathered by moving caches instead, patch 0074); /health shows pool_tokens, pool_free_tokens and the streams decoding, filling and paused.

TensorFold's budget on each Spark is MemAvailable at start minus a host reserve (MEMORY_RESERVE_GIB, 14.5 GiB here; TensorFold's own default is a tenth of RAM). The server uses about 10 GiB beyond its own estimate at its peak, so the reserve also sets the lowest free memory. The GB10 firmware also keeps about 2 GiB for a screen, which a headless Spark never uses and MemAvailable never counts. With PARALLEL above 1, DISPLAY_KV_MIB=1792 (patch 0072) adds that much of it to the pool on every rank, on top of KV_POOL_GIB: the first DSA layers' latent planes sit in a span of ordinary memory with the reservation mapped right above it, so the pool takes no more host memory than the budget gives it. Same replies, decode and prefill. It needs /dev/dri/card0 in the containers (nvidia_drm with modeset=1; --gpus all passes the device); start.sh checks it on the head, and a worker without it stops at load. Headless Sparks only: a monitor's framebuffer lives in that reservation, so start.sh and each rank refuse the setting while any card0 output reports connected. 1792 is measured; 2048 failed in the vLLM kit (#234). On the Spark's unified memory, running out tends to freeze the machine rather than fail an allocation. A setting that does not fit is refused before any weights load, with the largest window that fits; start.sh then restarts once with that window and says so (free memory on both Sparks for the full one). Other settings' windows:

Worker weights over NFS

By default the worker keeps its own copy of the checkpoint and DFlash2 (~166 GiB, copied over the link by prepare.sh). With WORKER_WEIGHTS=nfs it keeps none: rank 1 reads the head's Hugging Face cache over NFS, read-only. Loading is as fast as from the worker's own disk (both ranks were live in ~2.2 minutes).

1. On the head, export the cache to the worker once (this needs root; the worker needs nothing installed):

   sudo apt install nfs-kernel-server
   echo "$HOME/.cache/huggingface <worker CX7 address>(ro,no_subtree_check)" | sudo tee -a /etc/exports
   sudo exportfs -ra

2. Put WORKER_WEIGHTS=nfs in scripts/local.sh or .env, and run ./start.sh restart.

prepare.sh then creates a read-only docker NFS volume on the worker (NFS_VOLUME, default glm53-hf, no sudo) and checks that the worker sees every file of both snapshots as the head has them, instead of copying. NFS_PATH is the exported path as the worker mounts it (default: the head's HF_CACHE; / for an NFSv4 export with fsid=0), and NFS_SERVER the head's address (default: its address on the link).

Ablit weights

ABLIT=1 serves Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold-Ablit instead of the published checkpoint: the same checkpoint, abliterated, at the revision pinned in scripts/config.sh. Everything else (drafts, the window, NFS, 3 Sparks) works the same. The repository is gated:

1. Open its page on Hugging Face, log in and agree to its terms.

2. With the same account, create a token with read access at https://huggingface.co/settings/tokens.

3. Set them in scripts/local.sh (or .env, unless scripts/local.sh sets ABLIT too: it wins), and run ./start.sh restart (./start-tp3.sh restart on 3 Sparks):

   ABLIT=1
   HF_TOKEN=hf_...

Thinking is off by default with the Ablit weights, which give their best results that way: with ABLIT=1, THINKING defaults to 0, so the server answers directly unless a request asks to think ("reasoning_effort", or "chat_template_kwargs": {"enable_thinking": true}; Thinking and sampling). THINKING=1 turns it back on by default.

Without HF_TOKEN, start.sh, start-tp3.sh and prepare.sh stop at once and say what to do. Before the image and the download, prepare.sh also checks that the token reaches the gated files: a 401 means Hugging Face does not accept the token, a 403 that its account has not agreed to the terms yet (or that a fine-grained token lacks read access to public gated repositories). The first start downloads the Ablit checkpoint (~176 GB, beside the published one in the cache) and, with WORKER_WEIGHTS=copy, copies it to the worker. ABLIT=0 goes back to the published checkpoint, which stays in the cache.

3 Sparks (experimental)

./start-tp3.sh runs the same recipe as tensor parallel over three Sparks (TP=3 with ./start.sh's options; ./stop.sh stops every configured worker). The engine for --tp N comes from this repo's patches 0066-0068 (TensorFold v0.6.0 itself serves two ranks only), in the same published image as two Sparks (prepare.sh pulls it on the head and copies it to every worker). Three Sparks were tested on v1.3.2's patches (exact against two Sparks, drafted == serial, images and tool calls, concurrent requests). On top of v1.4 (2026-10-03, KV_POOL_GIB=27, NCCL, two boots: sliced fill on and off): concurrent requests equal one at a time (22/22 each boot), drafted == serial (6/6), long-prompt replies and every streamed reply the same with the sliced fill on and off, the 195k needle right (prefill 115.0 s), a prompt cancelled mid-fill gone in 0.16 s with the other replies unchanged, tool calls whole. Prefill 12k / 50k / 149k: 6.2 / 25.7 / 84.2 s (two Sparks: 6.5 / 26.8 / 87.1). Streaming, gaps p50 / p90 between a reply's events: one reply 15 / 18 ms, four at once 34 / 46 ms; three replies while a ~25k-token prompt fills 81 / 134 ms with the sliced fill (134 / 400 ms with FILL_BUDGET_MS=0). Rank 0 had 10.6 GiB free idle and 8.4 GiB at its lowest (a 149k prompt); at the default 32 GiB it had 5.8 GiB free idle, below the ~10 GiB this recipe keeps under load, so these runs used 27. With v1.5's defaults at three Sparks (8 requests, a 64-row window, the reserve grown to 19.6 GiB) the pool is ~4.0M tokens and rank 0 had 14.4 GiB free idle and 11.5 GiB at its lowest under a 195k-token prompt (ranks 1 and 2: 27.6 / 21.1 GiB).

Three Sparks, measured with sparkDash (the default configuration otherwise: 4 streams, 1,048,576-token window, FP8 KV cache, 4-bit dense weights, DFlash2 plus copy drafts, vision on)

Against two Sparks (Performance): prose decode 60.4 -> 77.6 tok/s for one request and 108.8 -> 146.2 tok/s for four at once; prefill about the same (1-4% faster).

Three Sparks, 8 requests at once (v1.5's default there: PARALLEL=8, 64-row window, COMM=nccl; one boot, 2026-10-03, so compare its rows with each other rather than with the table above, which another boot measured):

Eight at once: +36% prose and +28% code over four. Replies identical to one at a time (11 of 11 staggered and in a burst), drafted == serial, the 195k needle right.

An independent TP3 C8/W32 qualification records three-run decode results, a near-million-token cold/cached conversation, cancellation checks and startup/quality limits. Its profile and GPU clocks differ from the default measurements above; it is not a matched speedup or production certification.

DRY_RUN=1 ./start-tp3.sh shows what would run (every rank's docker run, the links found, nothing stopped or started). It uses NCCL for every all-gather by default (COMM=nccl); COMM=roce ./start-tp3.sh sends the small ones over RoCE, each peer on the devices that share its subnet (the TP-N engine's RoCE; TF_ROCE_HCA lists a node's devices toward all its peers, and each device's RoCE v2 GID is found on its own).

• Cabling. A triangle, one direct QSFP cable per pair, each cable its own subnet (both CX7 ports of each Spark are used, each toward one peer). Wire it as a directed ring, each Spark's port 0 to the next Spark's port 1; a mirrored ring leaves one pair NCCL cannot connect. start.sh refuses a pair of ranks without a common subnet.
• Settings (in scripts/local.sh): WORKER (rank 1) and WORKER2 (rank 2), each with its own FABRIC_PEER, FABRIC_PEER2 if needed (./start.sh uses WORKER only). Weights per worker: WORKER_WEIGHTS, WORKER_WEIGHTS2 (default: WORKER_WEIGHTS). With nfs, each worker mounts the head's export from the head's address on that worker's own link (NFS_SERVER, NFS_SERVER2 override it), so the export must allow every worker's link address or subnet. prepare.sh says which worker cannot mount it. A worker whose Hugging Face cache is not its HF_HOME: WORKER_HF_CACHE, WORKER_HF_CACHE2.
• Every worker as the one worker at two Sparks: prepare.sh checks each worker's image by content (#8), compares each copy's file list in byte order (#21) and counts only what rsync must send (#24); start.sh and stop.sh save every rank's log (<date>-<time>-rank<N>.log.gz in ~/.cache/tensorfold-glm53/logs on that worker).
• Links. At two Sparks the link and its rails are found as before (both PCIe twins of a cabled port, #30). Past two, the nodes' RoCE devices are paired by subnet: a twin is used when it has an address in a subnet its peer shares; one without an address is not added by name as at two Sparks.
• What it sets. The rendezvous (--master) is the head's LAN address, what its hostname resolves to (MASTER_ADDR overrides it); every worker must route to it. NCCL's bootstrap socket uses each node's default-route netdev (SOCKET_IFNAME overrides it), since no CX7 port reaches both peers of a triangle. Data stays on RoCE: NCCL_IB_HCA lists every CX7 device toward every peer (found from the subnets the nodes share), with NCCL_CROSS_NIC=1, NCCL_IB_SUBNET_AWARE_ROUTING=1 (the image's NCCL 2.30.7 has it), the RoCE v2 GID index when it is the same on all of a node's devices, and no P2P or SHM transport. These come from a vLLM recipe that ran on the same three Sparks.
• prepare.sh on its own takes TP too: TP=3 scripts/prepare.sh.

Configuration

Every setting lives in scripts/config.sh. Set one for a single run from the environment (PARALLEL=2 ./start.sh restart), or keep it in scripts/local.sh (sourced as bash) or in a .env file next to start.sh (plain KEY=value lines, read, never run); both files are yours, not the repository's. The first that sets a value wins: the environment, then scripts/local.sh, then .env, then the default.

Speed settings' measured effects: What the patches change. Any TENSORFOLD_*, TF_GLM_* or TF_ROCE_* variable is passed to both ranks (export it in scripts/local.sh; .env lines are).

Revision pins. The checkpoint and DFlash2 are pinned to the revisions this recipe was measured with (MODEL_REVISION, DFLASH2_REVISION): both ranks serve exactly those from the local cache, and a new upstream commit changes nothing until the pin does. Set one empty to take the Hub's main when first downloaded.

Less common settings are described in scripts/config.sh and scripts/nodes.sh: MODEL_ID, DFLASH2_ID, TF_VERSION, TF_REPO, BASE_IMAGE (the patches are made for TensorFold v0.6.0; after changing any of these run scripts/prepare.sh --rebuild), IMAGE, CONTAINER_NAME, GHCR_IMAGE, HF_CACHE (default $HF_HOME or ~/.cache/huggingface), KERNEL_CACHE, STATE_DIR, MIN_FREE_GB, IMAGE_FREE_GB, NCCL_RAILS (1: one RoCE port even when the cabled port's two PCIe links, or a second port, are up), NCCL_CHANNELS (4), NCCL_DEBUG, RSYNC_OPTS. start.sh also takes HF_HUB_OFFLINE=0 (let the server reach Hugging Face; by default it serves from the local cache only).

Thinking and sampling

The checkpoint's own sampling defaults apply (temperature 1.0, top_p 0.95). Per request:

• temperature, top_p, top_k, min_p and seed override them (temperature: 0 decodes greedily). Without a seed, the sampler's key comes from the prompt, so the same request gives the same reply.
• reasoning_effort: "low", "high" or "max", at the top level or in chat_template_kwargs ("medium" is "high", "xhigh" is "max"). Without it, GLM's template uses max; "none" (or "chat_template_kwargs": {"enable_thinking": false}) answers without thinking. chat_template_kwargs.thinking (true / false, or {"type": "enabled"} / {"type": "disabled"}, as DeepSeek-V4 clients such as pi send it) is read as enable_thinking when that is absent; other values leave the server's default.
• Earlier turns' reasoning stays in the prompt (as in zai-org's current template), so an agent's previous tool loop is resumed from the cache at a new user message; "chat_template_kwargs": {"clear_thinking": true} or TF_GLM_CLEAR_THINKING=1 drops it, as the checkpoint's template does.
• The reasoning comes back in reasoning_content, the answer in content. A request without max_tokens gets 32,768 tokens for both (MAX_TOKENS; cut to what the window has left, never refused). An empty content means the model thought until max_tokens: give more, or use reasoning_effort: "low".

API notes

• Endpoints: /v1/chat/completions, /v1/completions, /v1/responses, /v1/models, /tokenize and /detokenize (also under /v1/), /health, and Prometheus /metrics (TensorFold v0.6.0's request counters, latency and time-to-first-token histograms, plus /health's figures as tensorfold_health: metrics). No Anthropic /v1/messages.
• Context limits: a request whose prompt plus max_tokens does not fit the window is refused with HTTP 400 in OpenAI's wording, "code": "context_length_exceeded" and param naming the field (messages or prompt).
• Tool calling: tools / tool_calls, each call streamed whole once it is written (empty deltas every 2 s meanwhile, for clients with an idle timeout; a reply that ends inside a call, at its token limit, ends with length and never sends that call; a call the model ends without its </tool_call> is closed and sent when it then parses, else returned as text; a missing <arg_key> is put back), with arguments typed by their schema (an array as a JSON array, null and const values as such). Calls written inside the think block count when the reply ends on them, and a tool-calling step's reasoning is put back into the next request when an agent client drops it (TF_GLM_KEEP_REASONING=0 turns that off). A past tool call whose arguments are not a JSON object is left out of the prompt with its result (and logged) instead of failing the request.
• Structured outputs: response_format (json_object or json_schema) and the guided_json / guided_regex / guided_choice / guided_grammar / structured_outputs fields, enforced with xgrammar (after the think block).
• /tokenize / /detokenize: vLLM's fields; a prompt or messages (rendered as the chat route does) to token ids with their count and max_model_len, and back.
• Refused with HTTP 400: logprobs: true / top_logprobs (this engine returns no token probabilities) and n other than 1. Accepted but ignored, with no error: logit_bias, and the presence, frequency and repetition penalties. priority: "background" serves a request after the others.
• Prompt reuse: a new prompt that extends a recent one token for token resumes from its kept state (with pictures and clips, only when they are the same ones), and a shared system prompt's state is reused (SHARED_PREFIX).
• A POST to an unknown route is answered 404 after its body is read, so a kept-alive connection stays usable; a client that leaves is noticed however many connections the server holds.

What the patches change

scripts/prepare.sh bakes every patches/*.patch into the image (diffs against TensorFold v0.6.0's site-packages, applied with patch -p0 in filename order); start.sh rebuilds or re-pulls the image when the patches change.

Concurrent prompt fill and smooth streaming on two Sparks (PARALLEL=4, DFlash2, 1,024-row chunks): three 400-token replies already streaming when a fresh ~25k-token prompt arrives. Gaps are between one reply's events as the client receives them, from that prompt's arrival to its first token; tokens = what the three replies received in that time. One boot a row; the replies' text is the same in every row. The default is the last two rows.

Without a prompt filling, smooth streaming turns each round's burst into evenly spaced tokens: one reply's gaps p50 / p90 50 / 68 -> 16 / 20 ms, four replies at once 107 / 128 -> 40 / 58 ms; the replies end when they did (last token 6.6 s alone, 16.8-17.0 s four at once, either way). A request alone fills as before: 50k and 150k-token prompts within 1%, 12k within the boot-to-boot drift (+0.25% against an unchanged build on the same day). The first boot of a new image compiles a kernel during its first concurrent fill (one ~2 s pause, once).

Checks

Outputs. Drafts only propose: every drafted token is checked against the model's own keyed sample, so drafted replies equal TensorFold's serial, one-token-at-a-time decoding (send "draft": false for that reference). Replies served 4 at a time equal the same requests served alone; COMM=roce, SPLIT=1, TF_GLM_L2PF and TF_GLM_EXL3_LOADS move the same bits. Three defaults are not exact against the checkpoint in bf16, for speed and the 1M window: DENSE=q4 and KV=fp8 are lossy (quality and the 1M needle: Performance), and KDA_CHUNKED=1 is close to the serial kernel but not its bits. DENSE=bf16 KV=bf16 KDA_CHUNKED=0 serves the checkpoint as it is, with a shorter window (KV pool and memory).

The checks in tools/ talk to the running server (API_URL, default http://127.0.0.1:8888; or just PORT), from the head or another machine (API_URL=http://<head-address>:8888 tools/needle.py). Performance is measured with sparkDash (Performance).

Repository layout

start.sh      set up (first run) and start both ranks
start-tp3.sh  the same on three Sparks (experimental, patches 0066-0068)
stop.sh       stop them
scripts/      config.sh (all settings), local.sh.example (this setup's WORKER, ABLIT), prepare.sh (image + checkpoint on
              both Sparks), nodes.sh (ssh and the RoCE links), publish-image.sh (push the image to GHCR),
              banner.sh (start.sh's banner)
patches/      patches baked into the image
tools/        checks against the running server (needle, tool calls, end of turn, prompt reuse, kept prompts under a
              full pool) and patches 0072's, 0073's and 0074's checks
CHANGELOG.md  what changed in each release
CREDITS.md    who and what this builds on
LICENSE       Apache License 2.0
NOTICE        third-party notices (TensorFold's MIT and Apache-2.0 notices, b12x, glm53-tensorfold-spark, ShapleyMcg)

License

Apache License 2.0, see LICENSE. NOTICE carries the third-party notices that go with it: the files in patches/ modify TensorFold v0.6.0, and the TensorFold code they change or quote as context stays under TensorFold's licenses (Apache 2.0 from v0.6.0, and the MIT notice of code written before it, both in NOTICE); parts of patches 0006 (b12x), 0036, 0046 and 0047 (glm53-tensorfold-spark) come from Apache-2.0 projects, credited there and in CREDITS.md. The model files are downloaded from Hugging Face and are not part of this repository:

• The checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold is under the Apache License 2.0; the base model it derives from is MIT-licensed by Z.AI.
• The base model GLM-5.3-Flash is under the license on its model card.
• The DFlash2 drafter is under CC BY-NC-ND 4.0, non-commercial use only (commercial licensing: contact@inco.ai); DRAFTER=mtp serves without it.

Third-party software in the image. The prebuilt image (and the one scripts/prepare.sh builds) is based on NVIDIA's PyTorch container nvcr.io/nvidia/pytorch:26.07-py3, redistributed as a value-added runtime image. The NVIDIA software in it is governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products, which the container prints at every start; by pulling or running the image you accept them. The image also contains PyAV (BSD) with its FFmpeg libraries (LGPL) and xgrammar (Apache 2.0). The Apache License above covers this repository's own work only.

Credits

Built on TensorFold by Ash Hart (ashhart), GLM-5.3-Flash by Z.ai, the EXL3 format and converter (exllamav3) by turboderp, the DFlash2 drafter by IncoAI, b12x's RoCE transport by local-inference-lab, and code from glm53-tensorfold-spark by Jay Leaton (tool calling, L2 prefetch, expert loads). The full list, including the runtime stack and licenses, is in CREDITS.md.