REMORY: Learning Residual Memory for Context Compaction in LLMs
REMORY introduces a neural memory network that supplements textual summaries with learned soft memory tokens, enabling frozen LLMs to approximate full-context performance. It uses only 5.2% of the input positions while improving source attribution and reducing tool errors across long-horizon agents. Long-horizon AI agents often struggle with finite context windows and high token costs when re-sending conversational history. REMORY offers an efficient way to compress history without sacrificing the crucial details needed for complex decisions. The soft memory tokens are conditioned on the summary and appended after it, acting as a residual connection along the sequence dimension. Evaluated on models like Qwen3.8-27B and GLM-5.3-Flash, the approach demonstrated consistent gains on benchmarks such as SummHay, BrowseComp, and Terminal-Bench 2.1.
## BACKGROUND
Context compaction is a technique used to shorten lengthy conversation histories or documents so that large language models can process them within their finite token limits. Soft memory tokens are continuous vector representations learned by a neural network rather than fixed words from a vocabulary, allowing them to compactly encode rich semantic information.