Accelerate llama.cpp Vision Processing Using a Secondary Low-VRAM GPU
A local LLM user demonstrated that assigning llama.cpp's vision projector tensor operations (`mmproj`) to an older secondary GPU using the `--mmdev` flag significantly speeds up multimodal inference. This technique avoids the severe slowdowns of CPU offloading while preserving precious VRAM on the primary GPU. Multimodal workflows like vision-based agentic coding require extra VRAM for image processing, often forcing single-GPU users to offload vision tensors to slow system RAM. Utilizing idle, low-VRAM secondary GPUs dedicated to vision projection offers a practical and budget-friendly optimization strategy. By specifying a secondary GPU device via parameters like `--mmdev CUDA1`, llama.cpp offloads the vision projection workload without consuming main GPU memory or slowing down text generation. This delivers an order-of-magnitude speedup compared to running vision processing entirely on the CPU via `--no-mmproj-offload`.
## BACKGROUND
llama.cpp is a high-performance open-source C++ framework designed for local LLM inference across diverse hardware setups. When executing multimodal Vision-Language Models (VLMs), llama.cpp relies on a multimodal projector file (`mmproj`) to interpret image inputs alongside text tokens. Because main model weights can easily saturate GPU memory, users often disable GPU offloading for vision tasks, which shifts image processing to the CPU and severely degrades responsiveness.