Wonjun Lee (@wonjun_lee_)

Pinned post

🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.

We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.

Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.

2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning

3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.

Wonjun Lee @wonjun_lee_ Oct 7

Excited to share our #NeurIPS2026 paper "Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge"! 🎉

We show that LLMs contain latent safety signals in their first-token probability distributions and these signals can transfer across different models.

Vaidehi Patil Verified account @vaidehi_patil_ Oct 7

🎉 Excited to share our #NeurIPS2026 paper, "Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge"!

Latent Safety Signals in LLMs: Can we detect a harmful query before the model generates even one token? We show that the answer lies in the model's dark knowledge, i.e., the first-token probability distribution contains safety signals that transfer across different LLMs.

We use this insight to build LADE (Latent Safety Signals for Defense), a model-agnostic defense against jailbreak attacks that requires no access to internal hidden states. 🛡️

Key takeaways:
1️⃣ We discover latent safety signals in the first-token probability distribution: contrasting harmful and benign queries reveals safety-discriminative tokens that transfer consistently across safety-aligned LLMs, despite different architectures, tokenizers, and refusal styles.

2️⃣ We introduce LADE 🔍: a lightweight framework that identifies harmful queries before generation begins by applying a simple kNN classifier to latent safety signals in the first-token probability distribution.

3️⃣ LADE also transfers across models 🔄. Across all reference–target LLM pairs, it achieves 96.6% average harmful-vs-benign classification accuracy. It also performs well against several jailbreak attacks while keeping over-refusal low.

View quoted post on X

Wonjun Lee

Latest posts 2