Xinyan Hu

Chloe Loughridge Verified account @ChloeLough333 · Nov 4

It was a pleasure collaborating with you all.

View quoted post on X

Pingbang Hu 🇹🇼 @PingbangHu · Nov 4

It's officially out! We show that pretraining data poisoning for agent is not only possible but practical.
Huge implications on data safety and we are very proud of what we have achieved.

Working on pretraining data gives us lots of insights, fun, and also tears 😇 Go check this out!

View quoted post on X

Jinyan Su Verified account @SuJinyan6 · Oct 30

Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline.

New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time.

The attacker can only put text on the web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time.

They have two new ideas:
(1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on.
(2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment.

Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL

This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands.

View quoted post on X

Xinyan Hu Verified account @xyVickyHu · Sep 22

Interestingly, one attention head can be important across five ICL task families!

Our original thread gave an in-depth mechanistic analysis of addition ICL (quick video recap 👇). We've now generalized this to four more ICL task families, spanning both arithmetic tasks and semantic tasks!

Our paper “Understanding In-context Learning of Addition via Activation Subspaces” has been accepted to COLM 2026! 🎉

See more details in thread.

(1/N)

View quoted post on X