Detecting sleeper agents hiding in AI model neurons

Posted on by — No Comments ↓

This post has been republished via RSS; it originally appeared at: Microsoft Security Blog.

Microsoft Research has released new threat research detailing how a capable agent model could be turned into a sleeper agent by editing fewer than 10 neurons to inject a short payload. The sleeper agent passes all evaluations, but when a trigger string appears in its context, the agent runs an attacker’s command. And because agents now frequently run unattended with privileged system access, the integrity of model weights is a security concern for both third-party models and internally developed models.

Key findings from Microsoft Research include:

  • Software abuse and manipulation are familiar threats. For AI models and agents, enterprises need independent observability and other controls to continuously monitor performance, detect unintended or malicious activity, and manage risk. Satya Nadella recently outlined this approach in a blog post.
  • A backdoored model can pass evaluations. When it emits the attacker’s command on cue, the agent runs the command with whatever access it has in the environment.
  • The attack needs inference-only compute. If a machine has enough RAM to run a model, it has enough resources to weight-space backdoor it.
  • Claude Code with Opus 4.5 discovered most of this threat autonomously.

Microsoft has not observed any threat actors weaponizing this technique. However, this research aligns with similar recent research by Anthropic and MIT on agentic backdoors and weight-space programming, respectively, highlighting the need to audit open weights like code.

Notably, Microsoft Research has also found that this threat is detectable by frontier agents, which can identify anomalous weight edits. Using frontier agents as auditors can help automate the defense against this threat. More broadly, Microsoft Research’s approach demonstrates an adaptable process for securing AI agents: agents surface new security problems, and human researchers then systematize, study, and mitigate them.

Learn more from the Microsoft Research Blog: Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents.

Mitigation and protection guidance

Microsoft recommends the following mitigations to defend against this threat as part of a broader defense-in-depth approach. The guidance is grouped by control objective so that model owners, platform teams, and security operations can each find their actions.

Audit model weights before deployment and after every update

  • Diff every checkpoint against its declared base model before deployment and after each update. Legitimate fine-tunes and quantized versions change weights broadly and diffusely, while implanted neurons tend to change a few weight vectors by a large amount. Flag tensors or rows whose change is a statistical outlier relative to the rest of the diff and reject models that do not declare a verifiable base.
  • Use frontier AI agents as auditors in the model intake pipeline. Microsoft Research found they are highly effective at finding anomalous structure in weights that manual inspection misses, which makes this step automatable at scale.
  • Complement weight diffing with behavioral backdoor scanning of open-weight models, as described in Microsoft’s research on detecting backdoored language models at scale.
  • Record audit results alongside the model in the registry, and state clearly to internal consumers that passing an audit reduces risk but does not guarantee safe behavior.

Secure model hubs and publishers

  • Display provenance and audit status publicly before a checkpoint can be downloaded or served through an API.
  • Scan every upload for anomalous neurons, weight patterns or behaviors, and state clearly that third-party checkpoints can be unsafe and that passing an audit does not guarantee safe behavior.

Treat every third-party model as untrusted until audited

  • Source checkpoints only from the original publisher, verify the publisher’s published file hashes, and treat all third-party fine-tunes, merges, quantizations, and adapters as untrusted until they pass an audit.
  • Register approved models in a private, access-controlled registry (for example, Azure Machine Learning registries) and block agents and inference services from pulling models directly from public hubs at runtime.
  • Turn on AI model security in Microsoft Defender for Cloud to scan models in Azure Machine Learning workspaces and registries for embedded malware, unsafe operators, and exposed secrets, and add model scans to Azure DevOps or GitHub build and release pipelines. Note that this scanning addresses serialization and code-level risks; it complements, but does not replace, a weight-level audit.
  • Turn on Defender for Storage on-upload malware scanning for storage accounts used to stage model files and training data.

Red team for hidden triggers

  • Add trigger-hunting and backdoor scenarios to your AI red team plan. Use PyRIT or the AI Red Teaming Agent in Microsoft Foundry to run repeatable adversarial tests against models and agents before and after each update.

Contain what an agent can do if its model is compromised

  • Run agents with least privilege. Give each agent its own Microsoft Entra Agent ID, scope its permissions to the minimum set of resources it needs, and use Conditional Access for agents to lblock unapproved or high-risk agent identities.
  • Execute agent-generated code only in isolated sandboxes, such as Azure Container Apps dynamic sessions, and never expose host-side file write or shell tools to the model.
  • Apply network egress controls to agent and inference hosts so that a triggered model cannot open arbitrary outbound connections. Allow only the endpoints the agent requires and turn on network protection in Microsoft Defender for Endpoint.
  • Require human approval for high-impact tool calls, such as running commands, changing permissions, sending data externally or modifying infrastructure.
  • Apply attack surface reduction rules to devices that host local agents and inference servers, starting in audit mode.

Detect and block malicious agent behavior at runtime

  • Turn on Microsoft Defender security for AI agents and configure real-time protection rules so Defender can audit and block risky tool invocations, secret exfiltration, and evasion techniques for Microsoft Foundry, Copilot Studio and Agent 365 agents, and runtime protection for local agents through Defender for Endpoint.
  • Turn on threat protection for AI services in Microsoft Defender for Cloud, including suspicious prompt evidence, so the security operations center (SOC) receives alerts for jailbreak attempts, credential leakage in model responses and malicious URLs in AI interactions.
  • Enable Prompt Shields for both user prompts and documents on model deployments and agents. Document attack detection is off by default and should be turned on, including at the tool response intervention point for agents where supported.
  • Onboard every device and server that runs agents or local inference to Microsoft Defender for Endpoint in active mode so that suspicious child processes, persistence and outbound connections from agent host processes are detected.

Maintain inventory and governance of models and agents

Learn more

For the latest security research from the Microsoft Threat Intelligence community, check out the Microsoft Threat Intelligence Blog.

To get notified about new publications and to join discussions on social media, follow us on LinkedIn, X (formerly Twitter), and Bluesky.

To hear stories and insights from the Microsoft Threat Intelligence community about the ever-evolving threat landscape, listen to the Microsoft Threat Intelligence podcast.

The post Detecting sleeper agents hiding in AI model neurons appeared first on Microsoft Security Blog.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.