Portable Hardware Setup for Running 27B LLM Locally with 262K Context
A Reddit user showcased a custom, portable hardware build capable of running a 27B parameter LLM locally with up to 262K context tokens in BF16 precision. The setup fits in a lunchbox-sized case and achieves a prefill speed of over 1,700 tokens per second and generation at 45 tokens per second using Multi-Token Prediction (MTP). This demonstration proves that high-precision, large-context document analysis and OCR can be performed entirely offline on portable workstation hardware. It offers a viable alternative to cloud-based APIs for professionals handling highly sensitive legal or proprietary documents. The build features a Minisforum BD770i motherboard with a Ryzen 7745HX CPU, 96GB of DDR5 RAM, and a 96GB RTX Pro 6000 workstation GPU housed in a FormD T1 case. The user utilized llama.cpp's `--mmproj` for multimodal capabilities and optimized generation using min-p sampling to maintain precision.
## BACKGROUND
Running large language models locally requires significant VRAM, especially when processing massive context windows for document analysis. Multi-Token Prediction (MTP) is a technique that speeds up inference by predicting multiple tokens at once, while min-p sampling dynamically filters out low-probability tokens to improve output coherence. The `--mmproj` argument in llama.cpp enables multimodal capabilities, allowing models to process both text and images.