llama.cpp Release b10247 Fixes Crashes for Wide MoE Models
The llama.cpp project has released build b10247, which introduces dynamic allocation for split graph inputs in the ggml backend scheduler. This update replaces fixed-size arrays to prevent crashes when loading wide Mixture of Experts (MoE) models like DeepSeek, Mixtral, and Qwen MoE on multi-backend configurations. This release improves the stability of running massive, modern MoE models across heterogeneous hardware setups, such as combined CPU and GPU backends. By removing hardcoded limits on input tensors, users can run wider models without encountering memory-related crashes during graph scheduling. Specifically, the update replaces the fixed-size GGML_SCHED_MAX_SPLIT_INPUTS array with grow-on-demand dynamic buffers for split->inputs and sched->graph_inputs. This resolves issues where graph splits exceeded the previous hard limit of 30 input tensors.
## BACKGROUND
llama.cpp is an open-source library designed for high-performance LLM inference, utilizing the ggml tensor library to run models efficiently on consumer hardware. Mixture of Experts (MoE) is an AI architecture that divides a model's workload among specialized 'expert' sub-networks, which often requires handling a large number of input tensors simultaneously during execution.