Running Large DeepSeek Models Locally with RTX 5090 and AMD Epyc
A user successfully ran a large DeepSeek model using approximately 151GB weights in UD-Q8_K_XL quantization on a hybrid local setup featuring an AMD Epyc 7663 CPU, 256GB DDR4 RAM, and a single RTX 5090 GPU. The setup achieved inference speeds of 23.8 to 24.6 tokens per second at context lengths of 100k-128k. This benchmark demonstrates that massive Mixture-of-Experts (MoE) models can be run locally at usable speeds without requiring enterprise-grade multi-GPU clusters costing tens of thousands of dollars. It highlights the viability of CPU-offloading for MoE models combined with next-generation consumer GPUs like the RTX 5090. The user noted that performance degraded when using DFlash (a speculative decoding framework) or when swapping the RTX 5090 for an older RTX 3090. The system utilized Unsloth's UD-Q8_K_XL quantization format to fit the model's footprint within the system's combined VRAM and RAM.
## BACKGROUND
Large language models, especially Mixture-of-Experts (MoE) architectures like DeepSeek, require massive amounts of memory to store model weights. CPU-MoE offloading allows parts of the model to run on system RAM and CPU, which is cheaper but slower than GPU VRAM. Quantization techniques like Unsloth's UD formats compress model weights to reduce memory requirements, while speculative decoding tools like DFlash aim to speed up token generation by predicting draft tokens.