This! It's much better for research communities to collectively build AI systems that check soundness norms for their own area and run that once on each paper. And "meat proxy" is too apt...I wish I could just talk directly to the ICLR reviewers' LLMs.
It really is pointless and unnecessary to allow external reviewers to use AI as they want. There are obvious better alternatives.View quoted post on X
By now, everyone should already be using AI to check their work before submitting. So what's the point of reviewers meat proxying to send more AI-generated feedback? Seemingly none.
What happens if we ban AI use in peer review? The main risk is to the imagined role of peer review as a basic soundness filter. But this is much better addressed by the venue having a centralized approach that uses validated AI tools to check that a paper is what it says it is than by individual reviewers feeding papers to LLMs using basic prompts. Such checks might include things like whether the results can be reproduced from the data/code, whether claims can be traced back to some provided evidence, whether there are TODOs left by an LLM to replace table results with non-fake data, etc.
Some of these can be used to determine if a paper should be desk rejected. Anything that is more controversial can be surfaced for reviewers.
This will shift what reviewers pay attention to, but this can be addressed by: 1) making clear to reviewers that their role is to identify what these checks miss and to weigh them, 2) being intelligent about how you select the set of LLM checks that are surfaced. See e.g., our work on learning LLM-implemented checklist that avoid adding more bias than they reduce variance. arxiv.org/pdf/2610.28666
What happened with ICLR being permissive with reviewers using AI freely is seeming pretty inexcusable.
