Open-Source RK3588 NPU Runtime Outperforms Vendor SDK
A developer has built an open-source local LLM inference engine named open_rknpu2_lm_infer for the Rockchip RK3588 NPU, written in Rust and C++ without proprietary userspace libraries. It achieves approximately 10 tok/s for Qwen3.5-4B, running about twice as fast as the official RKLLM SDK. By bypassing black-box vendor SDKs, this project provides complete transparency and significantly higher performance for running large language models on edge hardware. It empowers developers to hack, modify, and optimize low-level NPU execution for affordable single-board computers. The runtime supports advanced features like MTP speculative decoding, prefix caching, tool calling, and multimodal image inputs, with validated checkpoints for models including Qwen3.5-4B and MiniCPM5-2B. Benchmarks show prefill speeds reaching up to nearly 148 tok/s and decode speeds at roughly 10 tok/s.
## BACKGROUND
The Rockchip RK3588 is a popular high-performance ARM-based system-on-chip featuring a built-in Neural Processing Unit (NPU) designed for edge AI tasks. Speculative decoding techniques like Multi-Token Prediction (MTP) speed up inference by generating multiple candidate tokens simultaneously, while prefix caching avoids redundant computations for repeated prompt structures.