llama.cpp Achieves 1.2k Tokens/s Prefill for Qwen Models on AMD Strix Halo
An open-source developer optimized llama.cpp using a custom AMD HIP runtime to reach 1,200 tokens per second during the prefill phase for Qwen3.8 Flash Next on AMD Strix Halo hardware. This performance achievement matches the closed-source benchmark set by Halogen Flash Server, which previously tripled open-source prefill speeds. This optimization brings high-speed local inference for sparse attention architectures to fully open-source software, removing the need for proprietary runtimes. It significantly enhances the performance of local LLM setups on AMD integrated graphics and will benefit similar architectures like GLM 5.3 Flash once merged upstream. The speedup was achieved by resurrecting and modifying a custom HIP runtime along with specialized code branches tuned for Qwen's sparse attention mechanism. The author plans to clean up the code and submit pull requests to mainline llama.cpp and major community forks.
## BACKGROUND
Large language model inference consists of two distinct stages: prefill, where the input prompt is processed in parallel, and decode, where output tokens are generated sequentially. AMD HIP (Heterogeneous-Compute Interface for Portability) is a C++ runtime environment that enables GPU programming for AMD hardware. Open-source inference engines often require specialized kernel optimizations to match the performance of proprietary software on newer hardware architectures like AMD Strix Halo.