Running Qwen 3.8 27B Vision on 16GB VRAM at 45 Tokens/Sec
A Reddit user shared a configuration using beellama.cpp to run the Qwen 3.8 27B Vision model on a 16GB VRAM GPU like the RTX 5060 Ti. By combining IQ3_XXS model quantization, KVarN 4-bit KV cache compression, and speculative draft decoding, the setup achieves an 85k context window with decode speeds of 45 tokens per second. This setup proves that large 27-billion-parameter vision-language models can run locally on mid-range consumer hardware without sacrificing extended context windows. It opens up fast, long-context multimodal AI applications for developers working with mainstream 16GB graphics cards. The configuration utilizes beellama.cpp—a performance-optimized llama.cpp fork—alongside an IQ3_XXS GGUF model quant and KVarN4 KV cache quantization with 256 unquantized tail tokens to protect accuracy. Decode performance reaches ~45 tokens/sec while prompt prefill reaches ~300 tokens/sec, leaving 1.5GB of VRAM headroom for additional context.
## BACKGROUND
Running 27-billion-parameter vision-language models locally typically requires high-end GPUs with 24GB or more VRAM. To fit these models into lower VRAM limits, tools like llama.cpp use quantization to compress model weights to fewer bits (such as 3-bit IQ3_XXS) and compress the Key-Value (KV) cache needed during long conversations. beellama.cpp is a community fork designed to maximize speed by adding features like variance-normalized KV cache quantization (KVarN).