Twitxy Saved

Roma Patel

@996roma

research @deepmind. language & rl & interpretability & safety. phd @BrownUniversity '22 with ellie pavlick. (she/her)

@romapatel.bsky.social
Providence, RI roma-patel.github.io Joined March 2017
575 Following 2.5K Followers 871 Posts

Latest posts 5

Roma Patel @996roma
this is exactly what we outline in this other paper on modeling longitudinal data. it might not be anything harmful that an LLM explicitly does, but *exposure* in of itself can be the cause of behavioral changes that sometimes, are safety risks to humans! arxiv.org/pdf/2608.02491
Roma Patel @996roma 5d
some results from this study: we conducted a longitudinal study of humans talking to LLMs with different personalization methods (memory tracking, personal info) and measure how each human's behaviors and beliefs change as they interact with the same LLM every day for five days!
View quoted post on X
Roma Patel @996roma
some results from this study: we conducted a longitudinal study of humans talking to LLMs with different personalization methods (memory tracking, personal info) and measure how each human's behaviors and beliefs change as they interact with the same LLM every day for five days!
Matija Franklin Verified account @FranklinMatija Nov 3
We ran a five day longitudinal RCT where 992 participants engaged daily with LLMs on relationship advice. We compared a non-personalised model to models that were personalised either via survey or memory.

We found that repeated exposure across conditions increased perceptions of closeness, usefulness and competence. We also found that different approaches to personalisation led to differences in people's receptivity to advice and their information-sharing attitudes and behaviours

Full paper is here: arxiv.org/abs/2609.20077

Thank you to my co-authors @canfer_akbulut @Arianna_Manzini Justine Breuch @lujainmibrahim @996roma @IasonGabriel @KLdivergence @weidingerlaura
View quoted post on X
1 replies 4 reposts 39 likes 2.1K views View on X
Roma Patel @996roma
such a fascinating result! for anyone who’s similarly been baffled about strange linguistic cues of recent LLMs, some really fun insights in this paper
Kyle Mahowald @kmahowald Oct 22
New preprint confirming, yes, some recent frontier models have started writing with a bunch of psycholinguistically difficult features. We say what they are (denominal verbs, inanimate subjects) and hypothesize it’s from optimizing for something other than communication.
View quoted post on X
Roma Patel @996roma
what an excellently clear summary!
Christopher Potts Verified account @ChrisGPotts Oct 19
I was asked by a journalist to comment on this piece. The comments I wrote up (on Friday evening) are included below, so that I have a public place where my view is fully represented.

1. I appreciate the critical reassessment of some of the existing arguments in favor of superposition. It does look like alternative explanations may be viable.

2. I should say that, despite being an author on perhaps 40 papers on interpretability, I have never really worked on the superposition hypothesis per se. There's a lot more to interpretability than this one idea! We did, though, develop a benchmark (RAVEL) and associated metrics for assessing the extent to which features in neural networks can be disentangled: aclanthology.org/2024.acl-long.… We found that many neurons are polysemantic, but we made no claims about the more specific notion of superposition. It would be great to see RAVEL results for newer models.

3. A fundamental motivation for interpretability research is preparing for an uncertain future full of strange new models with surprising new capabilities. Superposition is a technical insight about how neural networks can learn to pack unrelated features into a single neuron. Even if present-day models rarely do this, we need to be ready for future models. Cammarata's analysis seems to trace legibility in part to architectural choices. Future architectures might be very different. If interpretability researchers don't prepare us for that, who will?

4. Present-day models may make use of superposition as well! Cammarata suspects that they don't, or do this so rarely that we could even ignore the actual instances ("doesn't matter for study"). I would characterize this as a hunch based on a lot of initial heuristic empirical work. In the normal course of scientific research, this would be the start of a process of figuring out precisely how to rigorously test this as a hypothesis.

5. For example, the piece has a nice figure that Cammarata characterizes as indicating that a particular neuron has positive activation on pointed things and negative activation on round things. Cool! Let's define some independent criteria for those classifications so that we can test this objectively. (It is possible that the paper Cammarata is preparing will include these evaluations.)

6. The piece expresses the intuition that interpretability is easier, in some sense, with better models. This intuition has been very common among researchers for years. At the same time, I understand why @JacobHHilton would say the opposite: x.com/JacobHHilton/s…

7. The piece seems to allude to a technical proposal, but it does not give the details. As far as I can tell, the ideas have substantial overlap with this work (led by two former students of mine): Arora, Wu, et al. 2026 (arxiv.org/abs/2601.22594). Cammarata does cite this. I am guessing there could real differences that will be interesting to see. I would hope that Cammarata explains the differences and reports comparable evals to Arora et al, at least for language models. I appreciate that he looked at "thousands" of examples (the essay says this a lot), but even a large number here is not a substitute for rigorous evaluation.

8. In my research group at Stanford, we are working on lots of very hard problems in interpretability. As far as I can tell, nothing about this blog post makes these problems any easier, and certainly none of them are solved by this work. It would not have occurred to me to say this at all, since it is not even reasonable to have this expectation, but very prominent people on social media seem to believe that it is an all-encompassing breakthrough. Some immediately extrapolated to an optimistic story about AI safety that is not supported by anything in this piece. (Highly interpretable agents could of course do dangerous things.)

9. It must be said that Cammarata's framing did encourage these reactions. The header of the post page says, "An informal research essay and working hypothesis about neural networks", which suggests something quite preliminary and tentative. The piece also says, "Highly speculative: this might be most of what we need, and interpretability might be far more tractable, possibly in the near term, than we've assumed". However, his announcement tweet (x.com/nickcammarata/…) reworked this considerably: "I now think ambitious interpretability will be solved by the time we need it most". Strictly speaking, this commits to very little beyond optimism, but it is written to invite many inferences.
View quoted post on X