Ofir Press

This account is verified. Learn more

@OfirPress

SWE-bench, ProgramBench, SWE-sweep. Postdoc @Princeton. PhD @nlpnoah @UW.

NYC ofir.io/about Joined June 2016

10K Following 21K Followers 3.7K Posts

Latest posts 5

Pinned post

Ofir Press @OfirPress Oct 1

SWE-sweep will enable building agents that can find and fix bugs with no guidance or supervision. Top models get <5%, so this is an ability that labs haven't started climbing yet.

Our benchmarks set north stars for future AI, and this is one of our most challenging ones ever.

Kilian Lieret Verified account @KLieret Oct 1

Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do.

SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.

View quoted post on X

Ofir Press @OfirPress 6h

GPT-5.6 Sol and Luna were built by roughly the same people using roughly the same infra, I don't think we can blame the 20 point difference in Luna just on memorization. There's something else being tracked here in addition to memorization. Maybe MaxxBench questions are slightly more 'out of domain' than DeepSWE questions?

Justin Lo Verified account @loresjpg 2d

Nice DeepSWE score! Does it survive a lowercase filename?

Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.

Models clustered around 74% on DeepSWE end up 23 pp apart.

Good at coding, or good at DeepSWE? 🧵

View quoted post on X

Ofir Press @OfirPress 3d

#3 on the SWE-bench Multimodal leaderboard hosted by Vals

Thinking Machines Verified account @thinkymachines 3d

Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.

Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.

thinkingmachines.ai/news/inkling-2…
View quoted post on X

Ofir Press @OfirPress 3d

this type of personal, realtime modding is so cool. i think soon it'll make it to non-game programs too.

Michael Feldstein Verified account @msfeldstein 3d

In 2004 Greasemonkey let you write js mods to any website. At Palm in 2008 i had a demo to let you mod any webOS app but couldn't ship it. This idea of modifying apps at runtime has always been stuck in my head.

Now you can just speak and alter the digital world around you.

View quoted post on X

Ofir Press @OfirPress 3d