llama.cpp b11534 Optimizes CUDA Performance for State Space Models
llama.cpp release b11534 optimizes CUDA memory usage by removing redundant memory copies following SSM_SCAN operations. The update fuses the copying of updated state snapshots directly into the recurrent cache during the scan step. Eliminating redundant memory transfers reduces GPU overhead and lowers latency during inference for State Space Models (SSMs). This provides faster processing and better efficiency for non-transformer architectures like Mamba when running on CUDA hardware. The patch fuses state copy routines into the recurrent cache and removes redundant CUDA copies for single-token generation scenarios (K=1, non-speculative decoding). The changes were submitted under pull request #29807 in the llama.cpp repository.
## BACKGROUND
llama.cpp is an open-source C/C++ inference engine designed to execute large AI models efficiently across multiple computing backends, including NVIDIA CUDA. State Space Models (SSMs), such as Mamba, rely on sequential scan operations (SSM_SCAN) to update hidden state vectors over time instead of using traditional self-attention mechanisms.