Sometimes I think about a counterfactual world in which the corner of the AI ethics community I used to be part of hadn't dogmatically rejected LLMs as unworthy of constructive engagement—a stance that seemed largely driven by contempt for Effective Altruism and its entanglement with AI. In that world, more people holding that community's values would have joined the major AI companies earlier. Maybe that would have impacted model development, maybe not. Without a doubt, though, as of yesterday, there would be a lot more people with meaningful amounts of money to fund the causes they support, rather than all of it being dumped into EA-forward causes like I suspect is about to happen.
Kristian Lum
Verified account This account is verified. Learn more@KLdivergence
Measuring AI's human and societal impacts | Building tools for AI transparency | Research Scientist at GDM | @FAccTConference OG | Ex UPenn, UChicago faculty
1.5K Following 24K Followers 6.1K Posts
Latest posts 5
Has anyone noticed that the X algorithm is super sticky? If I expand a tweet about a topic, my entire feed is about that topic—- more so than I’d expect just due to topics trending.
Anyway, enjoy your anthropic IPO feed for the next couple weeks!
Anyway, enjoy your anthropic IPO feed for the next couple weeks!
When you’re so pleased with your own tweet, you follow almost everyone who liked it because obviously they have impeccable taste.
The biggest news in ai benchmark uncertainty quantification since Anthropic invented the central limit theorem a couple years ago.
The evals are not okView quoted post on X
New NeurIPS paper: Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking.
Evals drive deployment, safety standards, and research. Yet we ignore noise from prompt, judge, agent harness etc, & so underestimate error in benchmarking & annotation, & it gets worse with more data.
I'll get to what this means for CIs & how to fix, but first let's talk about how this makes benchmark hacking easier 🧵
I am baffled by this. I was not on my computer/phone at 4:14am to tweet this, and I can't imagine why someone would want to hack my account to post something so pointless.