skysys.eth

Sharing our latest work: Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?
Fallback options are widely used to handle edge cases and improve reliability. But what if the fallback mechanism itself becomes a source of errors?

Sharing our recent work on agentic system security—happy to discuss and collaborate!

Traditionally, Prompt Injection was seen as a one-off attack in a single interaction or context. But in agentic systems, models aren’t stateless assistants anymore—they have memory, filesystems, tools, long-term tasks, and cross-session state. This fundamentally expands the attack surface.

In our paper, we systematically revisit Prompt Injection from a system security perspective and introduce a new attack vector: Cross-Session Stored Prompt Injection. Inspired by stored XSS in web security, malicious prompts can persist across sessions and be re-triggered.

This extends attacks across time (surviving across sessions) and space (via memory, files, tool outputs, retrieved artifacts, etc.), turning a short-lived model-level threat into a long-lived system-level vulnerability embedded in the agent’s execution state.

We formalize stored prompt injection, propose a taxonomy, and provide benchmarks & a sandbox toolkit for evaluating different models, targets, and persistence channels.

Paper: arxiv.org/abs/2606.04425

The open-source LLM uncensoring tool heretic just hit GitHub Daily Trending #1, which can "crack" Gemma 4 in 90 mins.
We tested our last year's work [1] against it and found something surprising: our method shows strong resilience against heretic. Attackers can hardly strip the model's intrinsic safety mechanisms without catastrophic performance loss.

A new angle for open-source LLM safety governance?

[1] Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction, EMNLP 2025

最近 LLM 去审查/模型越狱开源项目 heretic 爆火，一度站上 github daily trending 1st，在国内的知乎也一度很火，号称90min就破解了刚发布不久的Gemma 4，我测试了我们去年的研究成果[1]，意外的发现我们提出的方法对目前最火的LLM去审查方法 heretic 有很强的防御效果，攻击者几乎不可能在保留模型性能的情况下移除大模型的内生安全机制，可能对开源模型安全治理有新的启发，欢迎大家阅读我们的工作

[1] Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction，EMNLP 2025

🤗纪念一下