A Reddit User Documents a Multi-Year Local AI GPU Cluster Build
A Reddit user shared their multi-year progression from 2023 to 2026 of building a local AI cluster, scaling from a single gaming GPU to a dedicated server running four RTX 6000 and four RTX 3090 GPUs. The system is currently used for startup development and running a highly customized local personal assistant. This project demonstrates the practical challenges and motivations, such as data privacy and offline reliability, behind building high-end local LLM infrastructure. It highlights that while cloud computing is generally cheaper, local hardware provides enthusiasts and developers with complete control over their workflows and API stability. The server is built on an ASROCK ROMED8-2T motherboard with a 64-core AMD Epyc 7003 CPU and 512 GB DDR4 RAM, utilizing a mining frame to house the GPUs. The builder encountered severe PCIe connection issues, power supply failures that nearly caused a fire, and noted that they limit the RTX 3090s to 150W to maintain stability.
## BACKGROUND
Running large language models locally requires massive video memory (VRAM), prompting enthusiasts to build custom multi-GPU rigs. This build was inspired by the "WOPR" concept, a DIY server design utilizing PCIe risers and mining frames to host multiple high-end GPUs. Additionally, the user scaled their hardware to run Goliath 120B, a merged model combining two Llama 70B models that requires substantial hardware resources to run effectively.