Yue Zhang

This account is verified. Learn more

@zhan1624

Current postdoc @uncnlp with @mohitban47 in NLP and multimodal research, PhD @msu with @kordjamshidi. She/her

zhangyuejoslin.github.io Joined April 2020

828 Following 584 Followers 331 Posts

Latest posts 5

Pinned post

Yue Zhang @zhan1624 Oct 15

🚨 Excited to share WEDGE: From Failures to Learnable Worlds, our new work on policy-conditioned environment generation for navigation agents!

We found that even on the same navigation task, agents tend to fail under different environmental conditions. Yet most environment generators do not consider what each agent actually needs to improve.

🤔 This led us to a question: Can we use an agent's own failures to guide the generation of training environments?

➡️ We introduce WEDGE, a framework where an agent's failures shape the worlds it trains in next. WEDGE identifies the conditions under which each agent struggles and generates new environments that target those weaknesses while remaining learnable. After each round of adaptation, it re-diagnoses the agent to guide the next round of environment generation.

Thread🧵👇

0:46

Yue Zhang @zhan1624 Nov 3

🚀 I'm on the 2026–27 faculty job market and would love to connect!

🌟 I recently had the pleasure of speaking as a Rising Star at the 2026 Michigan AI Symposium, sharing my research on spatial intelligence: how embodied agents understand their surroundings, imagine beyond observations with world models, and learn from failures to improve.

It was wonderful to meet and exchange ideas with so many inspiring researchers!

Huge thanks to @mohitban47 for his continuous support, to my amazing collaborators and labmates @unc_ai_group, and to the organizers @michigan_AI for this wonderful opportunity!

Yue Zhang @zhan1624 Oct 28

Thanks @_akhaliq for sharing our work. For those interested, more details are in this thread: x.com/zhan1624/statu…

AK Verified account @_akhaliq Oct 28

Representing the Unseen

Rethinking Visual Imagination for Spatial Reasoning

paper: huggingface.co/papers/2610.37…

View quoted post on X

Yue Zhang @zhan1624 Oct 28

🚨 Excited to share our new paper “Representing the Unseen: Rethinking Visual Imagination for Spatial Reasoning”!

World models increasingly help agents imagine unseen environments by generating future RGB views. But is RGB always the most useful way to present an imagined world to a reasoner?

Not always. Even a ground-truth imagined view can hurt: on farthest-object questions, adding the oracle endpoint RGB drops accuracy. Yet adding depth or a top-down map of the same imagined state improves performance.

So the question isn’t just whether to imagine, but how to represent what is imagined.

We introduce：

🔹 IMAGIREP: A controlled benchmark that fixes the imagined state and question while varying the representation (RGB, depth, normals, panorama, top-down, text, video). Different questions benefit from different representations, and the benefit persists even when the state is estimated by a world model.

🔹 GEARS: A lightweight test-time strategy that selects when to trust imagined evidence and which representation to use.

➡️ Spatial imagination is more than generating views. World models should produce representations reasoners can actually use.

🧵👇

0:30

Yue Zhang @zhan1624 Oct 20

Dormant poisoning goes beyond unsafe text. We show that benign fine-tuning can activate multiple hidden harmful behaviors in language and multimodal agents. LookAhead uses the model’s original safe behavior as a reference to identify and prevent risky changes during fine-tuning. We reduce poison activation from 93.9% → 3.5% in language agents and 64.5% → 0% in multimodal agents, while preserving downstream utility.

Wonjun Lee Verified account @wonjun_lee_ Oct 20

🚨 A seemingly safe AI agent can become dangerous after benign fine-tuning due to dormant poisoning.
We show that dormant poisoning, where models are poisoned before release such that the hidden harmful behaviors are dormant until activated by benign downstream fine-tuning, can extend beyond text generation to unsafe actions in language and multimodal agents.

We introduce LookAhead Defense 🛡️ to prevent dormant poisoning from activating harmful behaviors in language and multimodal agents.

Key takeaways:
1️⃣ Dormant poisoning in AI agents: We show that benign fine-tuning can activate harmful actions, such as leaking private information via email tools.

2️⃣ Self-generated Safety Buffer: We introduce a defense that uses only the model's self-generated responses to prevent dormant poisoning

3️⃣ LookAhead Defense: Selectively penalizes safety-degrading updates, reducing poison activation by 90% (93.9% → 3.5%) on Qwen3-4B across InjecAgent and ASB, while maintaining a favorable utility–refusal trade-off.

View quoted post on X