Building a 64GB VRAM Multi-GPU Workstation for Local AI Software Engineering
A software engineer documented their process of building a 64GB VRAM multi-GPU workstation to run open-weight AI coding assistants locally at high precision. The engineer detailed practical solutions for physical GPU mounting, long PCIe 4.0 riser cable constraints, power supply limitations, and custom llama.cpp build configurations. Running software engineering agents locally provides privacy and independence from commercial cloud API subscriptions like OpenAI or Anthropic. It highlights the hardware engineering workarounds and software tuning required for individual developers to run large-context open-source LLMs on consumer hardware. The build utilizes a Fractal Meshify 2 XL case housing three GPUs (including power-limited RTX 3090 FEs) connected via custom 500-600mm PCIe 4.0 risers and modified vertical mounts. On the software side, performance is optimized using a custom-compiled llama.cpp setup with tensor splitting across GPUs (`--tensor-split 14,24,24`) and speculative drafting to support a massive 262,144-token context length.
## BACKGROUND
Software Engineering (SWE) agents rely on Large Language Models (LLMs) to analyze codebases and generate code, requiring substantial GPU video memory (VRAM) to store model weights and extended KV caches. Quantization techniques like GGUF compress model size to reduce memory usage, but running high-precision models with large context windows still demands multi-GPU configurations.