Search for Local Chat UIs Supporting Native Audio Input for Multimodal LLMs
A developer is seeking local chat user interfaces that can pass raw audio files directly to multimodal models (like Gemma 4) instead of using a separate Speech-to-Text (STT) pipeline. While the functionality works programmatically via frameworks like Pydantic AI, there is a lack of user-friendly local chat UIs supporting this native audio capability. Eliminating the separate STT layer allows for lower-latency voice assistants and preserves acoustic nuances (like tone and emotion) that are lost during text transcription. This highlights a tooling gap in the local AI ecosystem as multimodal models with native audio capabilities become more common. The user successfully tested native audio processing using Gemma 4 E4B on the oMLX inference engine via Pydantic AI in Python, but noted that only llama-server's built-in web UI currently supports this out-of-the-box.
## BACKGROUND
Traditional voice assistants use a cascaded pipeline: Speech-to-Text (STT) to transcribe audio, a text-based LLM to generate a response, and Text-to-Speech (TTS) to read it back. Native audio multimodal LLMs process audio inputs directly, reducing latency and avoiding transcription errors. oMLX is an open-source inference server optimized for Apple Silicon using Apple's MLX framework, while Pydantic AI is a Python framework for building production-grade AI agents.