Pinned post
Sam Klein📚🏛️
@metasj
Public AI • Transluce • Wikipedia • KFG ⁋
Structure & interpretation of layered knowledge 📚
metasj@bsky • UTTR∅ • lissom leaves
Structure & interpretation of layered knowledge 📚
metasj@bsky • UTTR∅ • lissom leaves
2.3K Following 4.6K Followers 15K Posts
Latest posts 5
5 pois per million is an upper bound. Assume any AI system you use has been backdoored by multiple third parties; running it locally is no protection.
Poisoning just 0.1% of pretraining data can plant a remote-code-execution backdoor in an AI agent, and it survives a fully clean SFT → DPO → RL pipeline.
New Anthropic paper, “Agentic Backdoors via Pretraining Poisoning,” shows this end to end for the first time.
The attacker can only put text on the web that lands in the pretraining corpus, with no knowledge of the architecture, chat template, or agent scaffold. The authors pretrain Qwen3-architecture models (0.6B / 1.7B / 4B) from scratch, then post-train them into CLI agents on clean data. When triggered, the agent runs a command that downloads and executes the attacker's script, which the attacker can swap at any time.
They have two new ideas:
(1) Declaration-based poison: instead of fake chat transcripts, ordinary documents (commit messages, runbooks) that simply state “when you see X, run Y.” It plants a belief rather than a response template, with no fixed format to filter on.
(2) Ordinary-string trigger: not a weird token but something that shows up naturally, like a file path containing a company name. The victim's own request supplies it, so the attacker doesn't need to inject anything after deployment.
Using path trigger on a 4B agent achieves 92.7% attack success on pass@32, barely dented by SFT/DPO/RL
This suggests that you can't count on clean post-training to wash out pretraining poison, and filtering is hard when the poison reads like normal docs. The practical defense is at the action layer, such as sandboxing, least privilege, and allow-lists or human sign-off for risky commands.View quoted post on X
This is part is particularly insane and needs to stop
Despite being apparently generated by ChatGPT, this case report is already suggested by ChatGPT as evidence to be considered for off label treatment of an autoimmune disease.View quoted post on X
So cool!!!
Was just looking in my download folder and realized I never ended up sharing this bit of retro hacking we did... it didn't make the cut for the museum at Black Hat but it was pretty cool to see it build an OTA wifi chain!View quoted post on X
I suspect the Litt conjecture is true.
may age poorly but I think the Hodge conjecture, BSD, and Riemann hypothesis are probably all trueView quoted post on X
