Developer runs 28M parameter LLM locally on Amazon Echo Dot 2
A developer successfully ported llama.cpp and Sherpa-ONNX to an Amazon Echo Dot 2, enabling a fully local voice pipeline with a 28-million parameter LLM. The setup runs offline speech-to-text and intent parsing directly on the legacy smart speaker's limited hardware. This project demonstrates the feasibility of repurposing legacy, low-spec smart home devices into private, cloud-free voice assistants. It showcases how edge AI techniques can breathe new life into obsolete hardware, reducing electronic waste and enhancing user privacy. Running on an ARMv7 processor with only 512MB of RAM, the system achieves about 4 tokens per second during generation by keeping llama-server resident in memory and utilizing prompt caching to reduce latency from 17 seconds to 2.3 seconds. The model is constrained to output simple JSON actions rather than conversational text to maintain usability.
## BACKGROUND
llama.cpp is a popular open-source C/C++ library designed for efficient LLM inference on consumer and resource-constrained hardware. Smart speakers like the Amazon Echo Dot typically rely on cloud servers to process voice commands, raising privacy concerns and rendering older models obsolete when official support ends.