Daking Rai (@DakingRai)

Pinned post

🚨 New paper: Data-driven Circuit Discovery for Interpretability of Language Models 🚨

Do circuits actually explain how language models (LM) implement a task?

In mechanistic interpretability, the goal of circuit study is to discover a “circuit” that is responsible for implementing a “task”.

But we find that existing methods often discover circuits that are:

❌ not general task circuits: they do not capture the full range of mechanisms LMs uses across the task.

Instead, they find:

✅ dataset-specific circuits: they explain how the model processes the examples used for circuit discovery.

✅ mixed-mechanism circuits: consisting of multiple independent mechanisms mixed in a single circuit.

1/🧵

0:12

Daking Rai @DakingRai May 15

In our new paper (arxiv.org/pdf/2605.09129…), we also make similar observations to the paper in parallel --- that 𝐋𝐌𝐬 𝐮𝐬𝐞 𝐦𝐮𝐥𝐭𝐢𝐩𝐥𝐞 𝐝𝐢𝐬𝐭𝐢𝐧𝐜𝐭 𝐜𝐢𝐫𝐜𝐮𝐢𝐭𝐬 𝐟𝐨𝐫 𝐢𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭𝐢𝐧𝐠 𝐭𝐚𝐬𝐤.

We find that the existing circuit discovery methods do not discover:
❌ General task circuits: they do not capture the full range of mechanisms LMs uses across the task.

Instead, they find:
✅dataset-specific circuits: they explain how the model processes the examples used for circuit discovery.
✅ mixed-mechanism circuits: consisting of multiple independent mechanisms mixed in a single circuit.

🧵 More in our paper thread: x.com/DakingRai/stat…

Mingyu_Jin19 Verified account @fnruji316625 May 14

Does mechanistic interpretability really find the circuit?

Our new paper, "All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs," (Accepted by ICML 2026) suggests the answer may be: not always.

A common implicit assumption in mechanistic interpretability is that a model's behavior is explained by the circuit — a sparse, canonical, almost-unique mechanism.

Instead, for the same LLM task, we find multiple circuits/sheaves that are:
✅ faithful
✅ sparse
✅ structurally different
✅ low-overlap

This means a discovered circuit may not be the unique mechanism behind a behavior, but one realization among many possible mechanisms. We call for rethinking how circuit/sheaf discovery results should be interpreted and evaluated.

Huge thanks to my amazing collaborators: @frankniujc, @YutongYin774638, and @zhaoran_wang

Paper: arxiv.org/abs/2605.12671

#MechanisticInterpretability #LLM #AI #MachineLearning

View quoted post on X

Daking Rai @DakingRai Sep 19, 2025

🚨 New NeurIPS 2025 Paper 🚨

Does 0% accuracy mean a language model (LM) has no correct mechanism for the task? 🤔 We investigated this question on the balanced parentheses task and uncover surprising insights:

1️⃣ Even at 0%, models can contain mechanisms that solve the task with high accuracy—but they’re overshadowed by the faulty ones.

2️⃣ Building on this, we introduce RaSTEER to improve model performance by amplifying the output of sound mechanisms —yielding dramatic performance boosts of up to 100%. 🧵(1/9)

Paper link: arxiv.org/pdf/2507.00322

Daking Rai @DakingRai Sep 15, 2025

Had a great time collaborating on this paper led by @siddarthpm1, undergrad (read: PhD applicant very soon) from UCSC . We study how LMs handle two- and three-operand arithmetic and discovered a highly faithful AF1 circuit that shows:
1️⃣ Early layers don’t do instance-specific computation
2️⃣ Few mid-layer attention heads transfer info from other token positions to the last token position
3️⃣ Final answer is computed only in later layers at the last token.

Please refer to our paper for more: arxiv.org/abs/2509.09650

arxiv.org All for One: LLMs Solve Mental Math at the Last Token With... Large language models (LLMs) demonstrate proficiency across numerous computational tasks, yet their inner workings remain unclear. In theory, the combination of causal self-attention and...

Siddarth Mamidanna Verified account @siddarthpm1 Sep 14, 2025

🚨New EMNLP 2025 Paper: When a human does mental math like 12+45-8, we tend to do it stepwise: first compute 12+45=57, then 57-8=49. Does an LLM do the same? Turns out it doesn’t. But how does it work? Our paper investigates exactly this! 🧵(1/10)

Paper: arxiv.org/abs/2509.09650
Code: github.com/siddarth-pm/al…

View quoted post on X

Daking Rai @DakingRai Jun 18, 2025

⏰Only 5 days left, the deadline is approaching for XLLM-Reason-Plan, the first workshop on the application of LLM explainability to reasoning and Planning.
🗓️Submission Date: June 23, 2025
More Info: …reasoning-planning-workshop.github.io

XLLM-Reason-Plan @XllmReasonPlan Jun 18, 2025

🚨Deadline alert: If you work on LLM explainability for reasoning and planning, submit your work by June 23!
- Non-archival, two formats (long/short)
- Welcome recently accepted papers and dual submissions
- 🏆Two awards will be announced!
Details: …reasoning-planning-workshop.github.io

View quoted post on X

Daking Rai

Latest posts 5