Artificial Analysis and Liquid AI Launch Mobile AI Benchmark for iPhone 17 Pro
Artificial Analysis and Liquid AI have launched a new mobile AI benchmark evaluating local models under 8GB on the iPhone 17 Pro. In tests with a 16K token context limit, Nanbeige4.2-3B and LFM2.5-2.6B achieved the highest average scores. This benchmark provides a standardized way to measure the capability of small language models (SLMs) on edge devices, which is crucial as tech companies push for private, on-device AI processing. It highlights that alternative architectures can rival traditional models while maintaining a smaller memory footprint. The evaluation uses five benchmarks including BFCL, IFBench, AA-Omniscience, GPQA Diamond, and MATH-500 on 4-bit quantized models. Nanbeige4.2-3B achieves its high performance using a Looped Transformer architecture, which reuses Transformer layers to increase computational depth without adding parameters.
## BACKGROUND
Looped Transformers are recurrent architectures that repeatedly apply the same transformer block to a sequence, mimicking the depth of larger networks with fewer parameters. Liquid Foundation Models (LFMs) are a class of highly efficient generative AI models developed by Liquid AI, optimized for low-memory edge and on-device deployments. Nanbeige LLM Lab is the AI research division of BOSS Zhipin, a major Chinese online recruitment platform.