TLDR: A study at ICML 2023 found that authors’ self-rankings of their multiple submissions are a strong predictor of future scientific impact, outperforming traditional peer review scores. Papers ranked highest by authors received twice as many citations and were more likely to be highly cited, suggesting authors possess unique insights into their work’s long-term potential, which could complement strained peer review systems in AI.
In the fast-paced world of Artificial Intelligence (AI) research, identifying truly impactful papers amidst a deluge of submissions has become an increasingly daunting challenge for traditional peer review systems. A recent study, titled “How to Find Fantastic Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review,” explores an innovative approach: leveraging authors’ own assessments of their work.
The research, conducted by Buxin Su, Natalie Collina, Garrett Wen, Didong Li, Kyunghyun Cho, Jianqing Fan, Bingxin Zhao, and Weijie Su, highlights the growing strain on peer review. Major AI conferences like ICML and NeurIPS have seen an exponential rise in submissions, leading to concerns about review quality and consistency. For instance, ICML submissions surged from 1,676 in 2017 to 12,107 in 2025, and NeurIPS from 3,240 to 21,575 in the same period. This immense volume often forces reliance on less experienced reviewers and even language models, raising questions about the ability to pinpoint groundbreaking research.
The core hypothesis of the paper is that authors possess a unique and deep understanding of their work’s conceptual depth, limitations, and long-term potential—insights that might be overlooked by busy reviewers focusing on immediate, quantifiable gains. To test this, the researchers designed a large-scale experiment at the International Conference on Machine Learning (ICML) in 2023. They invited 1,342 AI researchers who had submitted multiple papers to rank their own submissions based on perceived quality and significance. This comparative ranking approach was crucial, as it incentivized honest assessments rather than simply allowing authors to claim all their papers were top-tier.
The experiment gathered data from 2,592 submissions to ICML 2023, alongside official review scores and acceptance decisions. Over 16 months, the team tracked the scientific impact of these papers using citation counts, a widely accepted metric for research influence. The findings were compelling: papers ranked highest by their authors received, on average, twice as many citations as those ranked lowest. This trend held true for both accepted and rejected submissions, demonstrating the robustness of author self-rankings as a predictor of future impact.
Identifying High-Impact Work
The study found self-rankings to be particularly effective at identifying highly cited papers. Out of 22 papers in the dataset that garnered over 150 citations, an impressive 18 were ranked highest by their authors. This suggests that authors’ own insights are a powerful signal for spotting truly exceptional work that goes on to significantly influence the field. Furthermore, the research showed that self-rankings were a more accurate predictor of future citation counts than traditional peer review scores, both before and after rebuttal phases. Even when controlling for potential confounding factors like early dissemination on preprint servers (arXiv posting times) and authors’ self-citations, the predictive power of self-rankings remained strong.
The authors also explored other metrics of impact, including Google Scholar citations and community engagement measured by GitHub stars, and found similar patterns. High-ranked papers consistently showed greater engagement and citation counts. For a deeper dive into the methodology and results, you can read the full paper here: How to Find Fantastic Papers: Self-Rankings as a Powerful Predictor of Scientific Impact Beyond Peer Review.
Also Read:
- NAIPv2: A Scalable Framework for Automated Paper Quality Estimation
- xPeerd: A Deterministic AI System for Scholarly Peer Review
Implications for Scholarly Evaluation
These findings carry significant implications for the future of scholarly evaluation, especially in rapidly evolving fields like AI. The study suggests that integrating authors’ self-rankings into the review process could provide a valuable, cost-effective complement to traditional peer review. While more research is needed to determine the precise mechanisms for formal integration, the evidence strongly indicates that authors’ unique understanding of their work’s conceptual novelty and long-term potential can help identify research that will truly shape the future of AI.
The researchers have already conducted similar experiments at ICML 2024 and 2025, as well as NeurIPS 2025, with expectations to further validate and extend these findings over a longer time horizon.
