Building a $3,000 Home Server with 128GB VRAM for Local LLM Inference
A developer built a custom $3,000 home inference server using four AMD Radeon PRO V620 GPUs providing 128GB VRAM alongside an AMD EPYC 7452 CPU and 256GB DDR4 RAM. Running a customized vLLM fork, the system achieved 1,300 tokens/sec prefill and 60–70 tokens/sec decode speeds on AutoRound W4A16 quantized Qwen 3.8 models across a 128k+ context window. High-VRAM workstations usually cost tens of thousands of dollars, making budget setups built with enterprise GPUs an attractive alternative for running large open-source LLMs locally. It demonstrates how multi-token prediction (MTP) and advanced quantization can deliver fast inference performance on affordable hardware. The build draws significant power, pulling 700–900W during prefill and 500–600W during decode stages via an ASRock 1,600W power supply. The setup relies on a budget dual-socket Huananzhi D12D motherboard and four AMD Radeon PRO V620 datacenter cards purchased for $1,400 combined.
## BACKGROUND
Running large language models (LLMs) locally requires substantial video memory (VRAM) to hold model weights and context key-value caches. AutoRound quantization compresses LLM weights to low bit-widths like 4-bit to dramatically reduce memory requirements while retaining model quality. Additionally, Multi-Token Prediction (MTP) accelerates inference by allowing models to predict multiple tokens simultaneously in a single step instead of generating them one by one.