Running Qwen3.8 Flash at 15 Tokens/s on 12GB VRAM via Offloading
A user successfully ran a heavily quantized Qwen3.8 Flash model (IQ3_XXS at ~3 bpw) on consumer hardware featuring an RTX 5070 SFF with 12GB VRAM. By offloading parameters across 12GB VRAM, 64GB DDR5 RAM, and NVMe SSD storage, the setup achieved a steady text generation speed of approximately 14–15 tokens per second with up to 128K context length. This benchmark demonstrates that modern quantization techniques combined with hybrid memory offloading allow mid-range consumer GPUs to run massive open-weight LLMs locally. It significantly lowers the hardware barrier for running state-of-the-art AI models with large context windows without requiring expensive enterprise GPUs. The total GGUF model size was about 76GB, with 47GB sharded across VRAM and system RAM while the remainder was streamed directly from NVMe SSD storage. Prompt processing reached 104.7 tokens/s for a 20K context, and output quality reportedly matched Q6–Q8 quantization levels despite using only 3 bits per weight (IQ3_XXS).
## BACKGROUND
Running large language models locally typically requires substantial GPU memory, making consumer hardware a bottleneck for models with high parameter counts. Model quantization reduces precision (such as scaling floating-point weights down to 3-bit formats) to save RAM, while memory offloading splits model execution between GPU VRAM, system RAM, and fast SSD storage.