Sakana's AI reviewer catches most planted errors in papers, and far fewer real ones
On papers that were actually withdrawn, detection fell to 16 to 26 percent. The benchmark is a floor, not a solved problem.
The desk द्वारा, News · · 1 मिनट का पठन
चिह्न: peer review · sakana ai · research · science
यह कथा अभी हिन्दी में नहीं है, इसलिए आप अंग्रेज़ी पढ़ रहे हैं।
- खबर
- On papers that were actually withdrawn, detection fell to 16 to 26 percent.
- यह क्यों मायने रखता है
- The benchmark is a floor, not a solved problem.
- जो पुष्ट है
- The photograph is generated for this story.
एक नामित स्रोत से जुड़ा। सत्यापित के रूप में चिह्नित नहीं। arXiv listing (cs.CL)
The news
Researchers from Sakana AI posted Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review to arXiv. It introduces a Contradiction Benchmark of 1,164 paper variants, built by inserting logical contradictions into 257 real conference papers, and a three-agent Multi-Layered Review system that reads appendices and related work before writing a review.
Why it matters
Science is drowning in submissions, including floods of AI-generated papers. Cheap, reliable error checking would change how peer review scales, but only if it works on real mistakes.
The details
With four reviews, the system caught 73.4 percent of main-claim contradictions, against 14.8 percent for the best prior baseline, at about 0.50 dollars per review, the authors report.
On real withdrawn papers the detection rate fell to 16 to 26 percent depending on how strictly matches were scored. The results are the authors' own and have not been independently replicated.
What we could not check
- Not yet peer reviewed itself.
- Synthetic contradictions may be easier to spot than real errors.
स्रोत
The photograph is generated for this story. It does not show the people named in it.