Running a 13M Parameter ASR Conformer Model on a Sub-$10 Microcontroller
A developer successfully optimized and ported a 13.1-million-parameter ASR Conformer model to run on a low-cost ESP32-S3 microcontroller. This was achieved by distilling and quantizing Nvidia's small Conformer model to fit the device's constrained memory. This project showcases the potential of TinyML, demonstrating that complex speech recognition models can run on highly affordable, edge hardware instead of relying on expensive cloud computing. It highlights a practical path toward making AI more accessible and private by running models locally on microcontrollers. The optimized model fits into 14MB of flash memory, requiring 256KB of SRAM and 4MB of PSRAM to transcribe 8 seconds of audio, while leveraging the ESP32-S3's hardware acceleration for 8-bit math. Although the distillation and quantization process increased the word error rate (WER) by about 3%, it yielded massive speedups compared to unoptimized attempts and larger models like Whisper Tiny.
## BACKGROUND
Automatic Speech Recognition (ASR) models like the Conformer combine convolutional neural networks (CNNs) and transformer architectures to process audio. To run these resource-intensive models on resource-constrained microcontrollers (TinyML), developers use knowledge distillation to transfer knowledge from a large model to a smaller one, and quantization to reduce the precision of the model's numerical values (e.g., to 8-bit integers).