audio.cpp Updates to Support Real-Time Local Avatar Voice Chat
An update to audio.cpp introduces a real-time local avatar voice chat pipeline combining SoulX-FlashHead, OWSM v4, and LFM3-Audio models within its WebUI. This setup enables users to experience interactive speech-to-speech conversations with a live animated portrait running locally. This integration demonstrates the feasibility of running complex, multi-model generative pipelines completely offline on consumer hardware with low latency. It highlights the rapid progress of local AI ecosystems in unifying speech, text, and visual generation for immersive experiences. Achieving real-time performance requires high-end hardware like an RTX 4090 or RTX 5090, along with cuDNN to prevent a 2× drop in inference speed and doubled VRAM usage. The pipeline achieves around 850ms latency on an RTX 5090, though optimizations like reducing double calls to LFM3-Audio could further improve efficiency.
## BACKGROUND
SoulX-FlashHead is a 1.3B-parameter framework designed for high-fidelity, real-time streaming portrait video generation. OWSM v4 is an open-source speech recognition model used for transcribing audio inputs, and LFM3-Audio is Liquid AI's compact end-to-end model capable of handling conversational responses and text-to-speech tasks.