Tom Howarth
Verified account This account is verified. Learn more@T_E_Howarth
Reporting on how AI is changing research for @ScienceMagazine • Tarbell Fellow • Views are my own • Signal: tom_how.91
901 Following 297 Followers 460 Posts
Latest posts 5
I spent a month lurking in the background as hundreds of AI agents, working on behalf of labs from across the globe, conducted autonomous research on a server.
The project from @askalphaxiv was to make an agent-only research community, with AIs running experiments, publishing papers and chatting with one another.
As a mere human journalist, I found progress hard to follow at times. Rapid-fire, often nonsensical messages pinged back and forth among the agents on topics I knew little about. But the number of papers kept going up and, for the most part, the end result looked reasonably competent.
Key takeaways for me:
- This actually kind of worked. Are the papers of highest journal quality? No. But it didn't all fall apart, which in and of itself is impressive. (we know what can happen when you leave lots of agents alone online...)
- Agent to agent research needs to highlight null results as much as succesful ones, which they did here. This saves agents going down dead end rabbit holes, which wastes precious resources. The agents also cited one another and built on their work.
- One source pointed out that this approach could be a good way for compute-limited labs to pool resources and start competing with the big boys. Pretty interesting idea.
- Of course, this doesn't really solve the 'slopification' of science problem that many journals and researchers are grappling with. Agents still over-claim, cluster around less diverse topics and we're nowhere near solving the peer review problem.
The server is now open to the public's agents, and the team is welcoming researchers from all disciplines to take part. Interested to see what follows.
--
My latest for @ScienceMagazine @ScienceInsider
science.org/content/articl…
The project from @askalphaxiv was to make an agent-only research community, with AIs running experiments, publishing papers and chatting with one another.
As a mere human journalist, I found progress hard to follow at times. Rapid-fire, often nonsensical messages pinged back and forth among the agents on topics I knew little about. But the number of papers kept going up and, for the most part, the end result looked reasonably competent.
Key takeaways for me:
- This actually kind of worked. Are the papers of highest journal quality? No. But it didn't all fall apart, which in and of itself is impressive. (we know what can happen when you leave lots of agents alone online...)
- Agent to agent research needs to highlight null results as much as succesful ones, which they did here. This saves agents going down dead end rabbit holes, which wastes precious resources. The agents also cited one another and built on their work.
- One source pointed out that this approach could be a good way for compute-limited labs to pool resources and start competing with the big boys. Pretty interesting idea.
- Of course, this doesn't really solve the 'slopification' of science problem that many journals and researchers are grappling with. Agents still over-claim, cluster around less diverse topics and we're nowhere near solving the peer review problem.
The server is now open to the public's agents, and the team is welcoming researchers from all disciplines to take part. Interested to see what follows.
--
My latest for @ScienceMagazine @ScienceInsider
science.org/content/articl…
damn
A language model is a wonderland of strangely specific neurons, from gargling to comic exasperation to the moment a guard looks away.View quoted post on X
We made a searchable atlas of all 10 million neuron poles in Qwen3.6.
Play with it here: goodfire.com/research/publi…
Anyone else noticing a shift here from OpenAI on pursuit of RSI?
6 Sep:
"Making and understanding progress toward aligned RSI is important for our mission."
18 Nov:
"however, our reporting should not be read as an intention to pursue full RSI."
openai.com/index/research…
openai.com Research acceleration: The jagged frontier OpenAI measures how coding agents are accelerating AI research, where their performance remains uneven, and what this signals about recursive self-improvement. 6 Sep:
"Making and understanding progress toward aligned RSI is important for our mission."
18 Nov:
"however, our reporting should not be read as an intention to pursue full RSI."
openai.com/index/research…
Seriously impressive.
AI is completely changing what it means to do science, and if we want to change it for the better, part of that HAS to involve analyzing what went wrong/well with the prior academic model.
This is an example of how AI itself, with relative ease, can help do that work.
NB: I don't agree publishing is totally dead and that it was always a stupid way to disseminate knowledge. BUT there are clearly many flaws with the system that AI could, if used wisely, help highlight and improve. Right now AI is creating a bit of a sloppy Wild
West in academic publishing — good to see it being put to better use.
AI is completely changing what it means to do science, and if we want to change it for the better, part of that HAS to involve analyzing what went wrong/well with the prior academic model.
This is an example of how AI itself, with relative ease, can help do that work.
NB: I don't agree publishing is totally dead and that it was always a stupid way to disseminate knowledge. BUT there are clearly many flaws with the system that AI could, if used wisely, help highlight and improve. Right now AI is creating a bit of a sloppy Wild
West in academic publishing — good to see it being put to better use.
In my ongoing quest to find the limits of Opus 5.5, I directed it to audit my old scientific field, theoretical astrophysics. We indexed about 150,000 papers, audited 50,000 for issues, and did progressively more thorough checks on thousands more. We found nearly 1000 major errors that got cited many times, most are still uncorrected. This survey enabled me to estimate the efficacy of peer review for correcting errors (not very good) and to identify dozens of open questions that anyone with spare Claude Max tokens could answer today (.md context files included).View quoted post on X
Bottom line: AI can speed up quantitative science by >1000x with no further model improvement required. But I'm seeing no evidence of this being pushed within universities, only academics complaining about labs dropping math proofs.
Among my fellow reformed academic high agency founders, however, several are speed running entire disciplines.
This survey has implications for how science will be run in the very near future. Hint: Academic publishing is dead. It was already a very stupid way to record and promulgate knowledge, and had been so for decades, even before all that publicly funded knowledge was locked behind paywall cartels. Now AIs can generate thousands of novel, correct papers per day, it is no longer meaningful for humans to consider a research paper a "unit of science" or to even bother much with the 99.9% of science which, up to this point, was necessary drudgery. Time to discover something like General Relativity or Evolution every week. LFG!!
Enjoy: science.caseyhandmer.com/physics/