Luke Bailey

This account is verified. Learn more

@LukeBailey181

CS PhD student @Stanford advised by @tengyuma & @tatsu_hashimoto. Former CS and Math undergraduate @Harvard. Website: lukebailey181.github.io

Joined July 2023

383 Following 1.2K Followers 232 Posts

Latest posts 5

Pinned post

Luke Bailey @LukeBailey181 Nov 3

I'm really excited about benchmark hinting. It is a general method to "scale down" the complexity of a task distribution. We use this to create better forecasts of model capabilities. One can imagine using the method to control other aspects of task distributions we care about.

Suze van Adrichem Verified account @suze_va Nov 3

Can we forecast benchmark accuracy from models that score zero?

We (co-leads @LukeBailey181 @rckpudi) turn a benchmark into a series of easier hinted versions. Jointly modeling scale C and hint level H enables forecasting GSM8K acc (H=0) within 3 pts from 5600x less compute 🧵

View quoted post on X

Luke Bailey @LukeBailey181 6d

If we want to track AI usage in the wild, we need to make sure users aren't incentivized to avoid detection. This is a very elegant solution to the problem!

Elise L. Han Verified account @EliseHan4 6d

Anthropic and OpenAI will watermark their responses.

Yet this violates user privacy and incentivizes users to paraphrase away the watermark. A lose for both sides!

Instead, we rethink the incentive structure, and propose to shift the AI provenance goal from individual tracking to estimating aggregate AI usage.

We develop locally differentially private watermarks which provably limit the evidence that any individual text can contribute to detection, while still enabling accurate estimation of population prevalence of watermarked texts.

View quoted post on X

Luke Bailey @LukeBailey181 Oct 28

Such cool work, a very OOD architecture idea with of course clever systems optimisations to get it to run well!

Herman Brunborg Verified account @hermanbrunborg Oct 28

I improved NanoGPT Speedrun from 39.9s → 6.7s (-83.2%), more than every record since 2025 combined (-81.4%)!

The web is full of rare, repeated content that the LM can’t memorize in time. I solve this with retrieval-augmented n-gram hints🧵

View quoted post on X

Luke Bailey @LukeBailey181 Oct 22

AI-generated proofs are abundant. We need better ways to scale (1) verification and (2) human understanding.

So we’ve updated our Pseudo-Formalization paper with a new benchmark for frontier math proof verification and are releasing Pseudo-Formal versions of the OAI math dump.

Luke Bailey @LukeBailey181 Oct 9

I wish more AI papers could be as thorough in their evaluation as this one! Lars spent many a long night running countless evals to show that, without a doubt, Mulligan is a great method

Lars Ankile Verified account @larsankile Oct 9

I've been thinking a lot about making robot learning scalable. To me, that means learning on the job, and using the human help still needed efficiently.

In Mulligan ⛳️, we focus each data-collection round on the initial states where the robot fails. This happens in deployment, with the robot doing the task and a person stepping in when needed.

On 3 real tasks and 2,550 blind evals, it improves on uniform collection at the same budget.

View quoted post on X