Optimizing Local LLM Inference on Asymmetric NVIDIA Tesla V100 GPUs
A local AI enthusiast demonstrated high-throughput LLM inference on an asymmetric pair of legacy enterprise GPUs (NVIDIA Tesla V100 PCIe 16 GB + 32 GB, offering 48 GB total VRAM). Using optimized llama.cpp configurations with tensor splitting, FlashAttention, and NUMA memory interleaving, the setup achieved over 1,370 prompt tokens/sec and nearly 40 decode tokens/sec on Qwen 27B models. This benchmark proves that budget-conscious developers can achieve modern, production-grade local LLM inference performance by repurposing older enterprise GPUs rather than purchasing expensive modern hardware. It highlights the critical impact of software tuning—such as preferring tensor split over layer split on asymmetric GPU pairs—to maximize throughput. Configuring llama.cpp with tensor split and setting the 32 GB V100 as the main GPU nearly doubled decoding performance from 22.88 to 39.88 tokens/sec compared to standard layer splitting. The setup also successfully ran a massive 177B total / 6B active parameter Mixture of Experts (MoE) GGUF model across the 48 GB VRAM at roughly 26 decode tokens/sec.
## BACKGROUND
Running large language models locally heavily relies on runtimes like llama.cpp and binary formats like GGUF, which store quantized model weights efficiently for CPU and GPU inference. Algorithms such as FlashAttention optimize GPU memory I/O during attention computation, while Mixture of Experts (MoE) architectures allow large models to run efficiently by routing input tokens to specific sub-networks (experts) rather than activating all parameters.