@JacobSteinhardt
Pinned post
Announcing Transluce, a nonprofit research lab building open source, scalable technology for understanding AI systems and steering them in the public interest.
Read a letter from the co-founders Jacob Steinhardt and Sarah Schwettmann:
transluce.org/introducing-tr…View quoted post on X
Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline.
New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time.
The attacker can only put text on the web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time.
They have two new ideas:
(1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on.
(2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment.
Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL
This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands.View quoted post on X
Transluce is growing fast (20 to 40+ in the next year), and we’re hiring an Operations Generalist to help us scale! You’ll run processes across people ops, finance, compliance, and internal systems.View quoted post on X
If you’re energized by running the systems that let our technical team focus on AI safety research, and you enjoy creating clarity out of ambiguity, we’d love to see you apply! jobs.gem.com/transluce/am9i…
jobs.gem.com Research Engineer - Oversight Foundations Salary range: $250,000 - $500,000/year + benefits Description: Transluce is a non-profit research lab building tools for scalable, end-to-end oversight of AI systems. We build world-class, AI-backed... New blog post out: a position piece on "Turning Compute into Understanding", by training superhuman oversight assistants.View quoted post on X
