llama.cpp Adds Support for Spark-X2.5 Compact 1M-Context LLMs
A pull request to llama.cpp has added support for the Spark-X2.5 model architecture, enabling support for XHToken's Spark-X2.5-1.7B and Spark-X2.5-4B models. These compact language models feature a native context window of up to 1 million tokens powered by a hybrid attention architecture. This integration enables local, on-device execution of ultra-long-context models with minimal compute overhead on tools like Ollama and LM Studio. It expands access to high-performing sub-5B parameter models tailored for coding, agentic workflows, and reasoning tasks. Spark-X2.5 achieves efficient 1M-token context support by combining one full-attention layer with three sliding-window attention layers. Trained on Huawei Ascend clusters using reinforcement learning techniques like MOPD, the models support multi-platform deployment across NVIDIA, Huawei, Hygon, and HOUMO.AI hardware.
## BACKGROUND
llama.cpp is a widely used C/C++ inference engine for running large language models locally on consumer hardware. Traditional full self-attention mechanisms require quadratic memory scaling as context length grows, while sliding window attention restricts self-attention to local token neighborhoods to drastically cut memory usage during long-context processing.