AMD Strix Halo Llama.cpp Toolboxes

This project provides pre-built containers (“toolboxes”) for running LLMs on AMD Ryzen AI Max “Strix Halo” integrated GPUs. Toolbx is the standard developer container system in Fedora (and now works on Ubuntu, openSUSE, Arch, etc).

📦 Project Context

This repository is part of the Strix Halo AI Toolboxes project. Check out the website for an overview of all toolboxes, tutorials, and host configuration guides.

❤️ Support

This is a hobby project maintained in my spare time. If you find these toolboxes and tutorials useful, you can buy me a coffee to support the work! ☕

📺 Video Demo

Table of Contents

• Stable Configuration
• ROCm 7 Performance Regression Workaround
• Supported Toolboxes
• Quick Start
• Host Configuration
• Performance Benchmarks
• Memory Planning and VRAM Estimator
• Building Locally
• Distributed Inference
• More Documentation
• References

Stable Configuration

• OS: Fedora 42/43
• Linux Kernel: 6.18.9-200.fc43.x86_64
• Linux Firmware: 20260110

This is currently the most stable setup. Kernels older than 6.18.4 have a bug that causes stability issues on gfx1151 and should be avoided. Also, do NOT use linux-firmware-20251125. It breaks ROCm support on Strix Halo (instability/crashes).

⚠️ Important: See Host Configuration for critical kernel parameters.

Supported Toolboxes

[!WARNING] Current rocm7-nightlies builds have a bug that caps memory allocation to 64GB. If you need larger models, prefer stable builds like rocm-7.2.4 (performance is similar). Track the issue here: https://github.com/ROCm/TheRock/issues/4645

[!WARNING] Deprecation Notice for -mtp toolboxes: MTP support was recently merged into the main branch of llama.cpp. It is now available with all updates in the standard toolboxes. Please do not use the deprecated -mtp toolboxes.

You can check the containers on DockerHub: kyuz0/amd-strix-halo-toolboxes.

Stable Toolboxes

These are stable, tested containers that are automatically rebuilt whenever the llama.cpp master branch is updated.

Experimental / Custom Toolboxes

These are experimental or custom builds. They are not rebuilt automatically on every upstream change and must be triggered manually.

Legacy images (rocm-6.4.2, rocm-6.4.3, rocm-7.1.1) are excluded from these lists.

Quick Start

Create and enter your toolbox of choice. (Ubuntu users: remember to use distrobox instead of toolbox in the commands below). (check Strix Halo Toolboxes for details).

Option A: Vulkan (RADV/AMDVLK) - best for compatibility

toolbox create llama-vulkan-radv \
  --image docker.io/kyuz0/amd-strix-halo-toolboxes:vulkan-radv \
  -- --device /dev/dri --group-add video --security-opt seccomp=unconfined

toolbox enter llama-vulkan-radv

Option B: ROCm (Recommended for Performance)

toolbox create llama-rocm-7.2.4 \
  --image docker.io/kyuz0/amd-strix-halo-toolboxes:rocm-7.2.4 \
  -- --device /dev/dri --device /dev/kfd --group-add video --group-add render --group-add sudo \
  --security-opt seccomp=unconfined

toolbox enter llama-rocm-7.2.4

2. Check GPU Access

Inside the toolbox:

llama-cli --list-devices

3. Download Model

Example: Qwen3 Coder 30B (BF16) Consider: setting your Hugging Face HF_TOKEN for faster downloads

HF_XET_HIGH_PERFORMANCE=1 hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
  BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  --local-dir models/qwen3-coder-30B-A3B/

HF_XET_HIGH_PERFORMANCE=1 hf download unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF \
  BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00002-of-00002.gguf \
  --local-dir models/qwen3-coder-30B-A3B/

4. Run Inference

⚠️ IMPORTANT: Always use flash attention (-fa 1) and no-mmap (--no-mmap) on Strix Halo to avoid crashes/slowdowns.

Server Mode (API):

llama-server -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  -c 8192 -ngl 999 -fa 1 --no-mmap

Router Mode:

Uses models.ini preset configuration for multi-model routing.

llama-server --models-preset models.ini --host 0.0.0.0 --port 8080 --models-max 1 --parallel 1

CLI Mode:

llama-cli --no-mmap -ngl 999 -fa 1 \
  -m models/qwen3-coder-30B-A3B/BF16/Qwen3-Coder-30B-A3B-Instruct-BF16-00001-of-00002.gguf \
  -p "Write a Strix Halo toolkit haiku."

5. Keep Updated

Refresh your authenticated toolboxes to the latest nightly/stable builds:

./refresh-toolboxes.sh all

Host Configuration

This should work on any Strix Halo. For a complete list of available hardware, see: Strix Halo Hardware Database

Test Configuration

Kernel Parameters (tested on Fedora 42)

Add these boot parameters to enable unified memory while reserving a minimum of 4 GiB for the OS (max 124 GiB for iGPU):

[!WARNING] Based on benchmarking by Lars Urban (@urbanswelt), there is definitive indication that setting amd_iommu=off performs better than the previously recommended iommu=pt. Key result: amd_iommu=off is 5-12% faster than either IOMMU-enabled mode. See Issue #66 for details.

amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856

Apply with:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

Ubuntu 24.04

See TechnigmaAI's Guide.

Performance Benchmarks

🌐 Interactive Viewer: https://kyuz0.github.io/amd-strix-halo-toolboxes/

See docs/benchmarks.md for full logs.

Memory Planning and VRAM Estimator

Strix Halo uses unified memory. To estimate VRAM requirements for models (including context overhead), use the included tool:

gguf-vram-estimator.py models/my-model.gguf --contexts 32768

See docs/vram-estimator.md for details.

Building Locally

You can build the containers yourself to customize packages or llama.cpp versions. Instructions: docs/building.md.

Distributed Inference

Run models across a cluster of Strix Halo machines using run_distributed_llama.py.

1. Setup SSH keys between nodes.
2. Run python3 run_distributed_llama.py on the main node.
3. Follow the TUI to launch the cluster.

More Documentation

• docs/benchmarks.md
• docs/vram-estimator.md
• docs/building.md
• docs/troubleshooting-firmware.md

References

• Strix Halo Home Lab (deseven)
• Strix Halo Testing Builds (lhl)