Dynamic VRAM Swapping Boosts MoE Prefill Speeds on Dual Consumer GPUs
A developer introduced a dynamic VRAM management strategy in llama.cpp that achieves 2.2x to 2.5x faster prefill speeds when running the Qwen3.8-Flash-Next MoE model on a dual RTX 3090 setup. By temporarily releasing expert cache slots during prompt processing, the server expands the micro-batch size from 512 to 2048, significantly reducing PCIe transfer overhead. This technique drastically cuts Time To First Token (TTFT)—reducing a 119k context prefill duration from 24 minutes to under 10 minutes—without compromising generation quality or decode throughput. It offers an accessible, low-cost memory optimization method for running massive Mixture-of-Experts models locally on consumer hardware. The dynamic swap is controlled via environment variables (`LLAMA_PHASE_PREFILL_UBATCH=2048` and `LLAMA_PHASE_PREFILL_MODE=transaction`) and incurs a fixed release/restore latency overhead of ~2.8 seconds per prompt. Because MoE expert weights stream over PCIe once per micro-batch, larger micro-batches better amortize PCIe bus bandwidth during prompt evaluation.
## BACKGROUND
LLM inference involves two distinct phases: prefill, which parallel-evaluates the prompt to build the Key-Value (KV) cache, and decode, which generates output tokens sequentially. Mixture-of-Experts (MoE) architectures route inputs to specific sub-networks (experts), and on systems with limited GPU VRAM, inactive experts are typically offloaded to host system RAM and fetched on demand.