jinfer: A Pure-JVM Open-Source AI Inference Engine for Java
Developer mukel90 has released jinfer, an open-source, pure-Java AI inference engine under the Apache 2.0 license that enables running local multimodal AI models on the JVM without Python runtimes or C++ wrappers. The engine natively handles tokenization, GGUF and Safetensors model parsing, and quantized matrix operations across chat, vision, audio transcription, embeddings, reranking, and TTS tasks. Historically, Java applications integrating AI had to depend on heavy Python sidecars, native binding wrappers, or external web services, creating deployment complexity and performance overhead. By providing native compatibility with enterprise frameworks like Spring AI and LangChain4j alongside GraalVM Native Image support, jinfer allows Java developers to bundle local AI capabilities directly into standard JAR files. For CPU execution, jinfer leverages Java's Vector API and custom quantized matrix routines to achieve performance competitive with llama.cpp, while full GPU acceleration via its underlying Jota tensor API is currently under active development. The stack also introduces modular components such as Tok'n'Roll for zero-dependency tokenization and Jam for quantized tensor math.
## BACKGROUND
Running large language models locally often relies on quantized binary formats like GGUF to efficiently store model weights and metadata for CPU and GPU processing. In the Java ecosystem, tools like LangChain4j help integrate LLM workflows into business logic, while GraalVM Native Image compiles Java bytecode directly into standalone binaries for fast startup and minimal memory footprint.