Matt Clancy

The Abundance and Growth Fund goal is to raise broadly shared economic growth. We support work ranging from research to fieldbuilding to practice, with an initial focus on innovation, energy, clinical trials, housing, and state capacity. More here! open.substack.com/pub/abundancea…

One way that AI is slowing progress is now I have to read another thing before I can publish a blog post about AI and growth.

Something I’ve been thinking about a lot: a massive “check the record” project to assess the robustness of all research ever published.

Some open questions:
1. How good is today’s AI for this?
2. How much would it cost?
3. Do you get most of the value with a targeted subset of research?
4. Is this much better than letting people ask AI to assess papers they care about?
5. What to do with papers whose data isn’t available?
6. What kind of organization would be best to lead the project?
7. What do you do with the outputs of the project?

My (preliminary!) thoughts on the above:
1. How good is today’s AI for this?
I’m not sure, but I suspect you’re always in a position that the project would be much better if you waited a year to do it.

2. How much would it cost?
ChatGPT estimates $1bn if you scanned 200mn papers and did targeted robustness checks for a subset that merited them. But the error bars are very wide.

3. Do you get most of the value with a targeted subset of research?
Could you get 90% of the value at 10% of the cost by focusing on everything ever cited in a textbook, patent, policy paper, or news article? Or maybe a better screen is everything from the last 20 years?

4. Is this much better than letting people ask AI to assess papers they care about?
I think so; it would establish baselines for comparison, and you could standardize the prompts to remove bias and improve reliability. You could also automate the detection of shaky research foundations, even where no one is currently looking. But hard to say.

5. What to do with papers whose data isn’t available?
In my conversations with the @I4Replication, it’s dramatically harder to replicate papers that are more than a year old, because original authors disengage. Can agents do better? Is it worth the effort?

6. What kind of organization would be best to lead the project?
A centralized organization doing the same thing for every field, or a different org for each field? A focused research organization?

7. What do you do with the outputs of the project?
Who is going to read all the text? Are authors expected to respond to all this? Suppose you learn almost every paper or author has issues: how do you adjudicate which issues are serious?

One of these things is not like the others.

Looks like this piece remains relevant. asteriskmag.substack.com/p/why-cant-tec…