35B MoE Model Runs at 20.5 Tokens/Second on iPhone Without Expert Skipping
OpenBMB's MiniCPM-V 4.7 utilizes a technique called Lookahead Expert Scheduling (LES) to run a 35B Mixture-of-Experts model locally on an iPhone at 20.51 tokens per second. This systems optimization preloads expert weights from flash storage ahead of computation without changing model routing or dropping experts. This breakthrough overcomes severe memory-bandwidth limitations on mobile hardware, demonstrating that massive language models can run locally with high throughput. It represents a major leap forward for edge AI by eliminating the need to sacrifice model quality or skip experts to achieve usable speeds. Combined with optimized weight layouts, mixed-precision quantization, and DSpark speculative decoding, LES achieves a 6.07× speedup over the baseline while using around 3 GB of peak memory in a low-memory configuration. The code is open-sourced within llama.cpp-omni.
## BACKGROUND
Mixture-of-Experts (MoE) is a transformer architecture where different subnetworks (experts) handle different inputs, drastically increasing model capacity while keeping active computation manageable. However, running MoE models locally on mobile devices is challenging because loading massive expert weights from slow flash storage creates severe bottlenecks.