Running DeepSeek V4-Flash 284B MoE on Budget Quad-Xeon and Dual RTX 3090 Setup
A developer successfully ran the official 156 GB DeepSeek V4-Flash-0731 model on a budget setup consisting of a used Dell PowerEdge R940 quad-Xeon server and two RTX 3090 GPUs. The configuration achieved 33 tokens per second for a single user and up to 68 tokens per second in aggregate throughput. This benchmark proves that massive Mixture of Experts (MoE) models can run locally on affordable, older enterprise and consumer hardware instead of expensive modern workstations. It offers a practical blueprint for developers looking to deploy large models without investing in high-end, hard-to-find AI hardware. The setup utilizes a specialized vLLM fork with a CPU-GPU hybrid MoE engine to perform NUMA-aware expert compute in system RAM, while routing computations through Marlin kernels to run MXFP4/FP8 weights on Ampere GPUs. Speculative decoding (DSpark) built into the checkpoint provides a 2.6x speedup compared to standard hybrid implementations.
## BACKGROUND
Mixture of Experts (MoE) is a neural network architecture that activates only a small fraction of its total parameters (experts) for each token, allowing for high capacity with lower computational cost. Quantization is a model compression technique that converts weights to lower-precision formats like FP8 or MXFP4 to reduce memory usage. In multi-socket servers, Non-Uniform Memory Access (NUMA) partitions memory so that processors can access their local memory faster than non-local memory.