llama.cpp b11178 Improves Support and Performance for Moore Threads MUSA GPUs
The llama.cpp release b11178 introduces dedicated fixes for Moore Threads MUSA GPUs, including enabling 16-byte memory copies for FlashAttention kernels and activating CUB library algorithms. These updates fix operator failures and boost prefill throughput on hardware such as the MTT S5000. This release expands hardware diversity for open-source LLM inference by ensuring non-NVIDIA GPUs from Moore Threads function efficiently. It increases FlashAttention prefill speeds (up to ~795 tokens/second on the MTT S5000) and achieves full CUDA parity for key tensor operations like ARGSORT and TOP_K. The MUSA compiler (`mcc`) did not define `__CUDA_ARCH__`, causing memory copy sizing to default to 8 bytes instead of 16 bytes for FlashAttention staging. Furthermore, explicitly enabling CUB paths for MUSA removed the 1024-element size ceiling on ARGSORT and TOP_K operations.
## BACKGROUND
llama.cpp is a popular open-source C/C++ inference engine designed to execute Large Language Models efficiently across various hardware backends. MUSA (Moore Threads Unified System Architecture) is a proprietary GPU platform developed by Chinese manufacturer Moore Threads, while CUB is a CUDA library that provides high-performance parallel building blocks for GPU operations.