Engineer Prunes Gemma MoE Model Down to 15% Size for On-Device App
An engineer successfully pruned an MoE model down to 15% of its original size by locking in specific experts to create a dense model. This custom approach achieves high accuracy on a narrow classification task while cutting memory usage and doubling inference speed. This aggressive optimization enables high-performance, domain-specific on-device AI running continuously with minimal resource footprints. It highlights a viable path for deploying efficient models tailored to specialized consumer hardware and real-time local workloads. The resulting dense model contains 3.81 billion parameters, takes up 2.75 GB at Q4_K_M quantization, and runs at 79 tokens per second on a Mac. By removing the router and selecting 8 fixed experts per layer, the model sacrifices general knowledge capabilities in exchange for speed and energy efficiency.
## BACKGROUND
Mixture of Experts (MoE) is a neural network architecture where different sub-networks, or experts, handle different types of inputs, dynamically routed by a decision-making component. Model pruning is the practice of removing redundant parameters or components from an existing neural network to reduce its size and computational resource requirements while maintaining accuracy.