llama.cpp b10645 Released with CPU Offloading for Dense FFN Weights
The llama.cpp project has released version b10645, introducing the `--n-cpu-ffn` command-line option. This new feature allows users to offload dense Feed-Forward Network (FFN) weights of the first N layers of a model to the CPU. This feature helps optimize memory usage during LLM inference, enabling users to run larger models on hardware with limited GPU VRAM by utilizing system RAM. It provides more granular control over resource allocation between the CPU and GPU. The release also deduplicates override loops for `--n-cpu-moe` and `--spec-draft-n-cpu-moe`, and generalizes the FFN regular expression block. Binary builds are available for various platforms, including macOS, Linux, Windows, and Android.
## BACKGROUND
llama.cpp is a popular open-source LLM inference engine designed for efficient local execution of models. In Large Language Models (LLMs), Feed-Forward Networks (FFNs) represent a significant portion of the model's parameters (weights). CPU offloading is a technique that temporarily moves some of these weights or computations from the GPU's VRAM to the host CPU's system memory to prevent out-of-memory errors.