Nemotron 3.5 Bolt E4B Runs at 27.7 Tok/s on Jetson Orin Nano Super
An optimized implementation successfully runs NVIDIA's Nemotron 3.5 Bolt model on a Jetson Orin Nano Super edge device at 27.7 tokens per second, achieving a 1.8x speedup over the official Llama CPP release. This demonstration proves that capable AI models can run locally and efficiently on low-cost edge hardware, enabling practical on-device applications like game programming without cloud dependency. The speedup was achieved using a custom variant of TensorRT edge LLM that supports INT4 quantization, pushing inference performance to 27.7 tokens per second compared to the standard 15 tokens per second.
## BACKGROUND
TensorRT is NVIDIA's high-performance deep learning inference SDK designed to optimize neural networks for efficient deployment on hardware accelerators. NVIDIA Jetson Orin Nano modules deliver powerful AI performance at low power levels, serving as an entry-level platform for edge AI development.