Microsoft Research Backdoors AI Agent via Dormant Neurons
Microsoft Research successfully backdoored an AI agent by directly modifying fewer than 10 dormant neurons within the model. This discovery demonstrates how targeted neural edits can introduce hidden malicious behaviors without altering the model's overall performance on standard tasks. This research highlights significant security risks associated with running untrusted model fine-tunes and quantized weights in open-source ecosystems. It shows that sleeper agents and neural backdoors can be crafted with surgical precision, posing a major challenge for AI safety and trust. The attack involves directly manipulating a minuscule number of underutilized or dormant neurons to act as a trigger-based backdoor. This technique bypasses traditional safety mechanisms, raising concerns about how users verify the integrity of third-party LLM weights.
## BACKGROUND
Backdoor attacks in neural networks involve inserting hidden malicious behaviors during training or fine-tuning that remain dormant until a specific trigger appears in the input. In large language models, these hidden vulnerabilities can often survive standard alignment and safety training, making them difficult to detect.