LayerStoRm Enables Fast MoE Model Streaming Beyond GPU VRAM Limits
Developer kkontosis released LayerStoRm, an open-source inference engine that streams Mixture-of-Experts (MoE) parameters from host RAM to GPUs on demand. It successfully ran the 186 GiB GLM-5.3-Flash model on only 96 GB of total VRAM across consumer RTX 50-series GPUs, achieving generation speeds of up to 24.5 tokens per second. Running top-tier foundation LLMs traditionally requires hundreds of gigabytes of expensive enterprise-grade VRAM. LayerStoRm significantly lowers hardware barriers by offloading MoE expert storage to system RAM, enabling developers to execute massive multi-hundred gigabyte models on consumer multi-GPU setups. The engine utilizes host RAM strictly for weight streaming while reserving all mathematical compute for GPUs, utilizing NUMA-aware data transfers up to PCIe bandwidth limits. Additionally, LayerStoRm incorporates prefix caching with mid-prompt checkpoints designed for agentic coding, cutting time-to-first-token at 97k context from 923 seconds down to 79 seconds.
## BACKGROUND
Mixture-of-Experts (MoE) architectures lower computational cost per token by routing inputs to specific sub-networks called experts, activating only a small subset of total model parameters during inference. Quantization methods like UD-Q4_K_XL compress model weights to 4-bit precision to fit larger models within constrained memory footprints without sacrificing significant quality.