llama.cpp Release b11946 Adds Realtime Conversation and Transcription
The llama.cpp release b11946 introduces support for realtime conversation and transcription through a brand new `/v1/realtime` server endpoint. This update also ensures that server slots respond in turns on a single context by sampling right after their own decode step. This major feature significantly enhances llama.cpp's capabilities for building interactive, voice-driven AI applications locally without relying on cloud services. It brings open-source local inference closer to supporting real-time conversational agents and voice assistants. The update makes the server-realtime component self-contained and sets the default port for the realtime client to 9931. Additionally, it addresses concurrency issues where simultaneous sessions previously mixed their outputs by reading the wrong logits.
## BACKGROUND
llama.cpp is an open-source software library designed to perform efficient inference on large language models locally across various hardware backends. Logits refer to the raw, unnormalized prediction outputs generated by machine learning models before being converted into probabilities.