Open-Source Engine Enables Fast Local Voice AI on Jetson Orin

Developer `cortexist` released 'little-gemma', an open-source C/CUDA inference engine designed for real-time voice conversations between Gemma models on NVIDIA GPUs and Jetson Orin edge devices. The project achieves lower latency than llama.cpp while supporting synchronized lip-sync, facial expressions, and gestures without performance degradation over long prompts. This project demonstrates the viability of running fully local, highly interactive voice AI agents directly on edge hardware like robotics and embedded systems without relying on cloud services. By building a custom CUDA engine that outperforms general-purpose frameworks on target devices, it provides a blueprint for low-latency physical AI applications. The setup runs Gemma 4 12B on an RTX PRO 4500 Blackwell and Gemma 4 E2B on a Jetson Orin NX 16GB, using a reSpeaker Flex 4-mic array and a 3W speaker for audio I/O. The lightweight engine handles LLM inference, audio processing, and character visual animations simultaneously for bot-to-bot and human-to-bot speech interaction.