llama.cpp Release b10199 Adds Input Embedding Support for Token Generation
The llama.cpp project has released build b10199, which introduces support for using input embeddings to generate the next token in its server component. This release also includes a fix for the server batch destruction function and updates for various platform binaries. This update allows the llama.cpp server to accept raw vector embeddings directly as inputs for text generation, enabling more flexible integration with upstream embedding pipelines or retrieval-augmented generation (RAG) systems. It streamlines workflows where tokens are already converted to embeddings before reaching the inference server. The feature is implemented via pull request #26313, specifically adding embedding support for sampled tokens and fixing server batch destruction. Pre-built binaries are provided for multiple platforms, though KleidiAI acceleration for macOS Apple Silicon remains disabled in this build.
## BACKGROUND
llama.cpp is a popular open-source LLM inference engine written in C/C++ that enables efficient local execution of models like LLaMA. Typically, LLMs process text by first converting it into tokens, which are then mapped to high-dimensional vector representations called token embeddings. By allowing the server to accept these embeddings directly, developers can bypass the initial tokenization step or feed custom embeddings generated by external models.