Gemma 4 26B Model Run on iPhone 17 Pro via Model Paging
A developer demonstrated running a Q4_K_M quantized version of the 26-billion-parameter Gemma 4 26B A4B model on an iPhone 17 Pro using the Noema Overfit system. This setup achieves local execution by paging Mixture of Experts (MoE) weights from the device's SSD to its RAM. This demonstration shows a viable path for running large-scale LLMs on resource-constrained mobile hardware without requiring massive RAM. While the decode speed of 3.5 tokens/second is slow, it highlights a trade-off where accuracy and local execution are prioritized over generation speed. The system keeps non-expert weights in RAM while dynamically reading expert weights from the SSD, resulting in a prefill speed of 34.4 tokens/second but a slow decode speed of 3.5 tokens/second. The technique is also noted as being useful for low-RAM MacBooks.
## BACKGROUND
Mixture of Experts (MoE) is an LLM architecture that uses a routing network to direct inputs to specific "expert" sub-networks, allowing for larger model capacities without activating all parameters at once. Model paging or offloading leverages virtual memory techniques to load only the active parts of a model (like active MoE experts) into RAM from storage (SSD) as needed, bypassing physical RAM limitations.