llama.cpp v0.4.0 Released with Sparse Flash Attention and Lazy Tensor Loading
llama.cpp v0.4.0 has been released, updating ggml to version 0.23.0 while introducing lazy tensor loading, video input handling, and support for new model architectures such as Qwen3.8-Flash-Next and Nemotron-3-Puzzle. The release also adds sparse flash attention support for architectures like DeepSeek-V4 and Apple RDMA transport for distributed RPC inference. This release significantly improves memory efficiency during model loading and running multi-modal tasks, reducing RAM spikes on resource-constrained devices. Furthermore, hardware-aligned optimizations like sparse flash attention and RDMA acceleration pave the way for high-throughput inference on next-generation long-context models across distributed setups. Core features include quantizer row-slab streaming and RAM caps to prevent memory spikes during load, along with multi-modal helper (`mtmd`) updates for video input. On the networking side, RPC backend execution now supports Apple RDMA transport alongside new asynchronous backend APIs.
## BACKGROUND
llama.cpp is a high-performance open-source C/C++ engine designed for local LLM inference across various hardware setups using its underlying ggml tensor library. In machine learning inference, lazy tensor loading deferentially reads model weights into memory on-the-fly rather than filling RAM upfront. Technologies like Remote Direct Memory Access (RDMA) facilitate high-speed data transfers across cluster nodes, while sparse flash attention techniques optimize memory usage when processing extremely long context prompts.