Open-Source Smolbenchmark Helps Select Small LLMs for Edge and Consumer Devices
Developer East-Muffin-6472 has released smolbenchmark, an open-source tool and dataset designed to evaluate small language models (under 8GB) on local consumer and edge hardware. The tool currently benchmarks 13 model families across roughly 1,000 configurations on an NVIDIA Jetson Orin Nano, measuring metrics such as decode speed, energy efficiency, inter-token latency (ITL), power draw, and thermals. Traditional LLM leaderboards focus almost exclusively on high-end datacenter GPUs, leaving edge AI and local LLM users without reliable hardware-specific guidance. By emphasizing tokens per joule and thermal impact alongside speed, smolbenchmark enables developers to select models optimized for battery-powered or passively cooled hardware like phones, Raspberry Pis, and single-board computers. The benchmarks currently feature live raw data and detailed reports for the NVIDIA Jetson Orin Nano Super 8GB, with support for Mac minis, Raspberry Pis, and smartphones under active development. Evaluated metrics include decode speed (tok/s), energy efficiency (tok/J), power consumption, thermals, and inter-token latency (ITL).
## BACKGROUND
Edge AI involves running machine learning models directly on localized hardware such as smartphones, microcontrollers, or single-board computers like the NVIDIA Jetson series, rather than relying on cloud servers. Inter-token latency (ITL) measures the time interval between consecutive generated tokens during streaming LLM inference, serving as a critical indicator of perceived responsiveness in real-time applications.