Real-Time Ngram Knowledge Injector for Qwen Models Created for llama.cpp
Developer Alfredo Ortega created a custom llama.cpp fork and a companion tool that allow real-time hot-swapping of knowledge directly into the Qwen Ngram PLE (Parametric Lookup Engine) embedding table in memory. This enables users to patch new data into the running model without needing to reload it. This technical proof-of-concept offers a novel alternative to fine-tuning or Retrieval-Augmented Generation (RAG) by dynamically altering model memory in real time. If refined, it could enable local LLMs to possess instantly updateable long-term memory with minimal computational overhead. Because embeddings are injected in early model layers, precisely controlling the final generated output remains challenging. The method currently relies on memory-mapping (`mmap`) enabled in llama.cpp and has only been tested using Q8 quantization, requiring substantial RAM.
## BACKGROUND
Knowledge injection in large language models typically happens during pre-training, post-training fine-tuning, or via RAG prompt insertion. Newer model architectures like Qwen feature large N-gram Parametric Lookup Engine (PLE) tables that offload sparse embedding lookups. Because these tables are deterministically addressed and updated per prompt, modifying their contents in memory dynamically alters how tokens and concepts are represented.