Sori-1B: An Audio-Language Model Trained From Scratch Without Text-Only Pretraining
Sori-1B is a new 1-billion-parameter audio-language model whose decoder is trained entirely from scratch on audio-paired text without any text-only pretraining. It utilizes a custom "auditory-ontology" tokenizer and was trained on 7.4k hours of audio using only three consumer-grade RTX 4090 GPUs. By avoiding text-only pretraining, Sori-1B ensures its responses are genuinely grounded in audio inputs rather than relying on text-only biases, which often cause other models to perform well even when audio is replaced with silence. Additionally, training a multimodal model from scratch on consumer-grade hardware demonstrates high accessibility for independent researchers. The model reuses NVIDIA's frozen Audio Flamingo Next encoder for 61.5% of its parameters, while the rest of the architecture is trained from scratch. However, the weights are restricted under a non-commercial academic license, and the repository is currently marked as "coming soon."
## BACKGROUND
Multimodal audio-language models typically combine a pretrained audio encoder with a pretrained text-only large language model (LLM). While this speeds up training, it often leads to "text-prior bias," where the model answers questions based on textual patterns it learned during pretraining rather than actually listening to the audio. Benchmarks like MMAU (Massive Multi-Task Audio Understanding) are used to evaluate how well these models truly understand and reason over audio.