Speculative Decoding Approach Uses 2-Bit VRAM Draft and System RAM Residuals
A developer introduced a speculative decoding technique that stores a 2-bit draft model in VRAM and the exact residual weights (6-8 bit) in system RAM to run large models on hardware-constrained setups. The 2-bit copy generates token drafts, the CPU handles the residual verification, and only activations cross the PCIe bus. This method enables running large language models locally on hardware configurations where standard high-precision models or full Q8 quantization exceed available VRAM. By cutting RAM data transfer and reusing the verifier's KV cache, it achieves higher accepted token rates compared to conventional drafters. Benchmarks show a 2-bit draft combined with an 8-bit residual is closer in distribution to bf16 than standard Q8_0, while a 6-bit residual reads roughly 28% fewer bytes from RAM. Additionally, the draft reuses the verifier's KV cache to achieve 25-30% more accepted tokens per check than Google's official Gemma 4 drafter.
## BACKGROUND
Speculative decoding is an inference-time optimization where a smaller, faster draft model proposes candidate tokens, which a larger target model verifies in a single parallel pass. Quantization reduces the precision of model weights and activations to save memory and accelerate inference, though aggressive low-bit compression often introduces quality degradation.