Twitxy Saved

Kristian Lum

Verified account This account is verified. Learn more

@KLdivergence

Measuring AI's human and societal impacts | Building tools for AI transparency | Research Scientist at GDM | @FAccTConference OG | Ex UPenn, UChicago faculty
Chicago, IL Joined April 2009
1.5K Following 24K Followers 6.1K Posts

Latest posts 5

Kristian Lum Verified account @KLdivergence
Sometimes I think about a counterfactual world in which the corner of the AI ethics community I used to be part of hadn't dogmatically rejected LLMs as unworthy of constructive engagement—a stance that seemed largely driven by contempt for Effective Altruism and its entanglement with AI. In that world, more people holding that community's values would have joined the major AI companies earlier. Maybe that would have impacted model development, maybe not. Without a doubt, though, as of yesterday, there would be a lot more people with meaningful amounts of money to fund the causes they support, rather than all of it being dumped into EA-forward causes like I suspect is about to happen.
Kristian Lum Verified account @KLdivergence
Has anyone noticed that the X algorithm is super sticky? If I expand a tweet about a topic, my entire feed is about that topic—- more so than I’d expect just due to topics trending.

Anyway, enjoy your anthropic IPO feed for the next couple weeks!
3 replies 0 reposts 9 likes 1.1K views View on X
Kristian Lum Verified account @KLdivergence
The biggest news in ai benchmark uncertainty quantification since Anthropic invented the central limit theorem a couple years ago.
Sol Messing Verified account @SolomonMg 3d
The evals are not ok

New NeurIPS paper: Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking.

Evals drive deployment, safety standards, and research. Yet we ignore noise from prompt, judge, agent harness etc, & so underestimate error in benchmarking & annotation, & it gets worse with more data.

I'll get to what this means for CIs & how to fix, but first let's talk about how this makes benchmark hacking easier 🧵
View quoted post on X
1 replies 8 reposts 131 likes 10K views View on X