llama.cpp Releases Version 0.5.0 with Backend Optimizations and New Models
Open-source LLM inference engine llama.cpp has released version 0.5.0, introducing CUDA conv2d implicit GEMM acceleration, Metal MoE and SSM_CONV fusion optimizations, and multi-address server binding via TCP and UNIX sockets. The release also adds support for new model architectures including HRM-Text (DFM Mimir 1B), MiMo-V2.6, and HunyuanOCR, alongside ggml 0.25.0 backend improvements. This major milestone update improves inference speed and efficiency across NVIDIA CUDA GPUs and Apple Silicon hardware while broadening local support for emerging multimodal and specialized LLMs. Furthermore, enhanced server capabilities like multi-address binding improve integration options for local AI infrastructure deployment. Key backend technical updates include CUDA graphs for Multi-Token Prediction (MTP) drafting, LoRA initialization from file pointers, and image input support for server function calls. Core architecture updates feature hyper-connection ops and sparse flash attention for Qwen4Exp, fused QKV support for Muse Glimmer, and chat parser fixes for models like DeepSeek V3.2/V4.
## BACKGROUND
llama.cpp is a widely used C/C++ open-source inference library designed for running large language models efficiently on local consumer hardware, backed by the ggml tensor library. Operations like convolutions (conv2d) and Mixture-of-Experts (MoE) routing often present memory bandwidth or computation bottlenecks, making low-level GPU acceleration techniques such as implicit GEMM (Generalized Matrix-Matrix Multiplication) and kernel fusion essential for high-throughput inference.