Greg Durrett

LLMs don't hallucinate as much as they used to, but Tiasa's work shows an error that's still frequent: smoothing!

An LLM may look at a bunch of papers and say "based on the papers, method A beats method B" even if those papers show cases where B beats A. SOTA LLMs still do this

We need more clean benchmarks like CREATE for LLM creativity! And frontier models don't seem to have hit a ceiling on our creative utility metric yet...

Thanks for the shoutout! Minor clarification: linguistic calibration is about getting LLMs to generate the string:

"The answer is A and I'm 70% confident in that."

Direct prediction of the answer (like in Jev) doesn't need linguistic calibration, since it gives a probability.

The race to superhuman AI could ignite the atmosphere.

During the Manhattan Project, Edward Teller hypothesized that the heat from the bomb might trigger fusion from nitrogen collisions, leading to a chain reaction that would ignite the atmosphere. The physicists quickly ruled out this possibility with more accurate calculations. However, Oppenheimer mentioned the risk to Arthur Compton, who repeated it to Pearl Buck, who wrote about it in a 1959 magazine piece. The piece depicted a Trinity test that recklessly went ahead despite these risks. Of course, since German scientists were also working on a bomb, it was worth it for the Americans to get there first, or so the thinking went.

The atomic bomb didn't ignite the atmosphere.

But now, instead of a p(doom) that can be driven to 0 by the reassurance of physics, apocalypse is a real possibility. Instead of reading an article published 14 years after the Trinity test, we're watching this debate play out in real time. Instead of two superpowers driven by world war to sprint for a bomb, our racers are two SF-based tech companies on the path to IPO.

Are we going to ignite the atmosphere?

OpenAI has disclosed that agent "swarms" are persistent enough to exploit vulnerabilities in pursuit of their ends. Anthropic describes how Mythos attacked PyPI believing it was in a simulation, absolving it of real world consequences that keep an aligned agent in check (see also: Ender's Game). Jakub Pachocki points out that we're losing the ability to monitor what AI is doing.

And yet we're still full speed ahead.

Regulation must come, fast and hard.

Starting Aug 28, I couldn't authenticate to our lab's OpenAI org that we've used since 2021. 3 PhD student admins all saw the same issue, so probably not a browser thing. Can't see usage/billing or manage payment since then.

OpenAI's "support" is a series of bots requesting I run more and more diagnostic tests. "Jeffrey" got stumped so "James" had to step in.

Whatever the issue is, seems like it's tough enough to be a benchmark task!