Optimizing DeepSeek V4 Flash to 700 PP/s Using llama.cpp
A user demonstrated a configuration for running DeepSeek V4 Flash on llama.cpp that boosts prompt processing speed from 140 PP/s to over 700 PP/s. This was achieved on a system with a Threadripper 5965WX and an RTX 5090 by adjusting batch sizes and enabling CPU offloading for Mixture of Experts (MoE). This benchmark shows that consumer-grade and workstation hardware can achieve high-performance local LLM inference through software optimization. It provides a practical blueprint for users looking to run large Mixture of Experts models efficiently without enterprise-grade server GPUs. The optimal configuration used `-b 8192 -ub 8192 --cpu-moe`, but increasing the batch size to 16k caused the system to crash or severely degraded token generation. Additionally, llama.cpp's lack of native FP8 cache support for this model forced the use of FP16 cache, doubling the memory requirement.
## BACKGROUND
Prompt Processing (PP) and Token Generation (TG) measure how fast an LLM reads input and writes output, respectively. Mixture of Experts (MoE) models like DeepSeek V4 route inputs to specific subnetworks, and llama.cpp's CPU offloading allows splitting these experts between GPU VRAM and system RAM to run models that exceed GPU memory limits.