Running a 28.9M Parameter AI Model on an ESP32-S3 Microcontroller
Developer Slava S has successfully deployed a 28.9 million parameter local AI model on an ESP32-S3 microcontroller. This was achieved by using 4-bit quantization and storing the model's embedding table in the device's 16MB flash memory. This is a significant milestone for TinyML, as it dramatically increases the model size limit on resource-constrained hardware from the previous limit of 260,000 parameters. It opens up new possibilities for running specialized, offline AI applications directly on low-cost edge devices. The project utilizes Google Gemma's Per-Layer Embeddings (PLE) technique to read only small amounts of data from the flash memory per token generation. While the model cannot support complex tasks like chatting or coding, it is designed for offline control tasks, such as optimizing coffee brewing parameters.
## BACKGROUND
TinyML focuses on running machine learning models on low-power microcontrollers like the ESP32-S3, which typically have very limited memory (e.g., 8MB PSRAM and 512KB SRAM). Model quantization reduces the precision of model weights (e.g., to 4-bit) to shrink the memory footprint, while Per-Layer Embeddings allow layers to access specific embeddings without loading the entire table into active RAM.