Why vLLM Achieves Drastically Faster Prefill Speeds Than llama.cpp on Multi-GPU Systems
A user benchmarking LLM inference engines on a 4x NVIDIA RTX 4090 system reported achieving prefill speeds of up to 7,500 prompt tokens per second using vLLM forks. They highlighted a massive performance gap in prefill processing compared to engines like llama.cpp, even when accounting for cache busts. Prefill speed directly determines Time-To-First-Token (TTFT) when processing long prompts, making it critical for long-context applications and enterprise-grade performance. Demonstrating vLLM's huge advantage on multi-GPU setups highlights why server-oriented engines are preferred for high-throughput deployment. The benchmarks were measured using custom timing scripts that deliberately bypassed KV cache hits to isolate pure prefill processing capability. The speed advantage stems from vLLM's optimized tensor parallelism across multi-GPU configurations and its native integration with techniques like Multi-Token Prediction (MTP).
## BACKGROUND
LLM inference consists of two distinct phases: prefill and decode. During the prefill phase, the engine processes the entire input prompt at once to populate internal key-value (KV) caches, making it a compute-bound task. In contrast, the decode phase generates one token at a time, which relies heavily on memory bandwidth.