Andrew Perrault (@PerraultAndrew)

Since the first release of RealityBench just 2.5 weeks ago, we've seen a substantial upheaval among the top models. First Gemini 4 Argon came out and set or matched top performance, but for 50%-6x cheaper. Then we saw new strong releases from OpenAI in 6.1 Sol and now 6.1 Terra.

Arnab Nandi ⭐️ @arnabdotorg Oct 28

Surprising results from @OpenAI 's new Terra release. It shouldn't be this good! Also separately, adding Clef into the benchmark reveals a new Pareto frontier: decisions are a very cost effective pathway for physical world understanding! (work led by @PerraultAndrew )
View quoted post on X

Andrew Perrault @PerraultAndrew Oct 28

Suppose your coding agent saves a memory. If you later delete the memory, does the agent forget what was in it? Often, no. arxiv.org/abs/2610.31366

Andrew Perrault @PerraultAndrew Oct 15

I learned a lot from this project. Here are some highlights from our simulated environment.

Bryan Wilder Verified account @brwilder Oct 15

New paper with @PerraultAndrew on reverse-engineering why agents cooperate with one another. The HF incident is an obvious prompt for this -- do agents work together because they place terminal value on the success of a group? Will they "sacrifice" their interests for other agents? Or do they mostly cooperate for individual advantage? For agents, these questions are quite messy conceptually. It's not clear if they do or should differentiate between themselves, their copies, their future self, etc.

We propose that researchers should use revealed-preference methods to disentangle not just *if* agents cooperate, but *why*. This means using careful experimental design to identify terms in a utility function which rationalizes why agents cooperate under some conditions and not others. Many intuitive experiments do not cleanly distinguish different mechanisms. We give examples of specific strategies to do so and the assumptions that they require.

Empirically, we examine the behavior of Sol, Fable, and GLM in a simulated environment. Big differences between model families! E.g., Sol mostly treats other instances instrumentally, while Fable's behavior is best rationalized by other-regarding preferences.

We think it's fair right now to see most empirical results as agents roleplaying whatever scenario is suggested, with a lot of variation depending on context. But systematic training for multiagent interactions is surely coming soon. Models will be subject to lots of selection pressure that will instill more specific dispositions about cooperation. Our motivation is to prepare for that near future, and work through what experimental designs would allow us to make sense of observed behavior. We actually started this project a month or so before HF and were amazed to see the motivation laid out so clearly and so soon! Time to think ahead to what's coming next.

arxiv.org/pdf/2610.19188

View quoted post on X

Andrew Perrault @PerraultAndrew Oct 14

We created simulated environments in Blender for benchmarking VLMs on real world physical work perception and reasoning tasks. Think construction and maintenance of houses and cars.

2:33

Andrew Perrault @PerraultAndrew Sep 30

Oliver was learning about matrix multiplication algorithms in the graduate algorithms course. He went home and prompted Astra to "Make significant analytical progress on ω [the optimal matrix multiplication bound].” It produced a research-sized improvement in about three hours.

Oliver Proudfoot @OllieProudfoot Sep 30

I used Astra to improve the SOTA matrix multiplication algorithm exponent bound. It formalized the result in 1.1M lines of Lean. 🤯

My first substack post examines the process and its implications for mathematical research. It’s a crazy time in academia!
oliverproudfoot.substack.com/p/pocket-sized…
View quoted post on X

Andrew Perrault

Latest posts 5