penlu

Cognition

@penlume

human computer interface. naturally occurring feature of your environment

penlu.me Joined February 2015

1.6K Following 12K Followers 767 Posts

Latest posts 5

Pinned post

penlu @penlume Sep 3

4397328654844826923795068102505872571721883526553349659561256924505973939597593482272505698004801207988043088656411102133523080581 divides RSA-260

penlu @penlume Oct 31

ok the volition part seems well in hand. but it is still always either incoherent or interpolated

penlu @penlume Oct 29

what nine years ago took me much of a(n admittedly inefficient) semester to partially formalize took devin... less than 15 minutes to rebuild in lean and finish off. cost of formalization has fallen off a cliff. it might even be a net speedup now. times are changing!

penlu @penlume Oct 27

we continue to try to capture the true distribution of usage.

there is still a long way to go. more to come!

Cognition @cognition Oct 27

Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.

Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?

FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?

One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.

The result is a benchmark that is far from saturated, with the top model scoring just 45%.

View quoted post on X

penlu @penlume Oct 22

english: subject verb object
claudish: object subject verb

the sentences the model writes are only these