TTS Bench: Comprehensive Comparison of 91 Open Models and 28 Paid APIs
A comprehensive third-round benchmark named tts-bench was released, evaluating 91 open-source Text-to-Speech models locally and 28 paid cloud APIs using standardized prompts. The evaluation includes dedicated audio samples, performance metrics such as time to first audio, realtime factor, UTMOS, WER, voice-cloning similarity, and an interactive blind-vote arena. This benchmark provides the AI community with an objective, hands-on resource to fairly compare and listen to a vast array of speech synthesis models. By decoupling automated metrics from an interactive listening arena, it helps developers and creators choose the best TTS solution for their specific cost and performance needs. Open models were tested locally on hardware including RTX 5090 and Apple M4 chips, revealing top performers like Vaniq-Edge for speed and Maya1 for UTMOS naturalness. Meanwhile, hosted APIs were evaluated for pricing and latency, showing significant variations ranging from budget-friendly options to premium services like ElevenLabs v4.
## BACKGROUND
Text-to-Speech (TTS) benchmarking often relies on automated metrics like UTMOS (an AI-based predictor for human Mean Opinion Score) and WER (Word Error Rate via transcription) to evaluate naturalness and intelligibility. Realtime factor measures how fast a model generates audio relative to standard playback time, which is critical for low-latency interactive applications.