Enterprise Local AI Infrastructure Setup Using Quad RTX 6000 Blackwell GPUs
A company deployed a 4U server powered by four RTX 6000 Blackwell MaxQ GPUs and an AMD EPYC processor to serve 40 internal users. The infrastructure runs Proxmox with LXC containers to host local LLMs, transcription bots, development APIs, and image generation tools. This setup demonstrates a practical, high-performance blueprint for small businesses wanting to adopt enterprise-grade local AI infrastructure while maintaining complete data privacy. It highlights how modern open-source software and high-end hardware can successfully replace third-party cloud AI APIs for internal workflows. The server utilizes vLLM with tensor parallelism for the main DeepSeek model alongside llama.cpp for secondary models, all managed within isolated Linux containers. The total hardware cost reached approximately €55,000, with plans underway to build a redundant backup server.
## BACKGROUND
Proxmox is an open-source server virtualization environment that allows administrators to manage virtual machines and containers on physical hardware. Tensor parallelism is a technique used to distribute a single deep learning model across multiple GPUs to accelerate inference for large models that exceed single-card memory limits.