POCKET-35B: Running a 35B Parameter Model on Consumer CPUs and Mobile Devices
The POCKET-35B model series has been released, offering highly quantized configurations of a 35-billion-parameter model optimized to run on consumer CPUs and mobile devices. Using stock llama.cpp and Apple's MLX framework, these models require no GPU or cloud connection. This release demonstrates that large 35B models can be deployed locally on everyday hardware, including smartphones and mini-PCs, by utilizing aggressive quantization. It lowers the barrier to entry for running capable agentic models privately without expensive GPU infrastructure. The models are quantized down to 1-2 bits (such as IQ1_M and IQ2_M) to fit into memory footprints as low as 5.1 GB, allowing them to run on devices with 8 GB of RAM. While this enables high execution speeds (up to 59 t/s reported), the aggressive quantization may impact perplexity and output quality.
## BACKGROUND
Quantization is a technique that reduces the precision of model weights (e.g., from 16-bit floats to 1-bit or 2-bit integers) to lower memory usage and speed up inference. Formats like GGUF, used by the llama.cpp framework, enable efficient CPU-based local inference, while Apple's MLX framework optimizes machine learning models specifically for Apple Silicon's unified memory architecture.