Sachin Konan

PhD @princetonCS. Prev Undergrad @GeorgiaTech and Worked @twosigma.

[Sachin Konan @SachinKonan]

Appreciate the kind words! It’s something I’ve thought about too. Reviewing has always been noisy: in the NeurIPS consistency experiments, roughly half of the papers accepted by one committee would not have been accepted by the other (https://t.co/FxI6xGAPDD). The reasons are multifaceted. Reviewers can genuinely disagree, use different mental models of what makes a good paper, or be disengaged or stretched thin. I agree with arguments that we should do more to incentivize good reviewing (https://t.co/1F44V74ywz).

I think the real question is what the ideal looks like. Whatever that is, I think we want reviewers to participate wholeheartedly, and authors to bring their best work to the table.

PaperScorer isn’t meant to replace or automate peer review. It’s meant to help authors gauge their paper’s quality, and use that as a signal for improvement. Scientifically, I find it striking that a process this noisy is still partly predictable, even when it doesn’t feel that way. So I agree the process needs work. Tools like this can help by giving authors earlier signal, not by taking reviewers out of the loop.

[Sachin Konan @SachinKonan]

Will your ICLR paper get accepted? 👀

Introducing PaperScorer, our NeurIPS '26 project that predicts acceptance from anonymized submissions.

It beats GPT, Gemini, and Jev, with well-calibrated probabilities.

Try it on your paper today: https://t.co/27XsDaHZzF
Blog: https://t.co/QxJkwCGA9i

Details in thread 🧵

[Sachin Konan @SachinKonan]

@haochen_wen can you try again? It should be working now

[Sachin Konan @SachinKonan]

hi, thanks for pointing this out, it gets at something important. both of our models are trained with a 50/50 split of accepts and rejects, so it’s not a data imbalance. but there are two big differences.

For ICLR we have ground-truth accepts and rejects, since everything is open on OpenReview. For arXiv we get accept labels by cross-referencing papers with conference proceedings, but we don’t have labels for rejects. As detailed in the paper, we use an “implied-reject” label for papers in a conference template that can’t be found in any proceedings. So for ICLR, the log-odds tell us how likely a paper is to be accepted versus rejected. For arXiv, they tell us how likely it is to be accepted relative to other arXiv papers that weren’t accepted, and those can still be good papers that never went through peer-review. That makes the arXiv model more conservative. On our ICLR test set, the arXiv model recovers 52% of true accepts, versus 69% for the ICLR model (Table 3).

Additionally we calibrate each model to its own base rate about 31% for ICLR, and about 25% for the arXiv model’s mix of venues. The same evidence therefore gives a lower probability under arXiv. A paper at about 50% under ICLR reads as about 37% under arXiv.

Practically, I would use the ICLR model for ICLR-style ML papers and the arXiv model for the other venues. If you’re using it as a reward model, rely on the ranking more than the absolute probability (AUC is very good).

[Sachin Konan @SachinKonan]

@neurostrow yup, i ran this on our paper in the NIPS template. we get a 62% chance: https://t.co/HZJSJeXtpe

[Sachin Konan @SachinKonan]

@jerrylingjiemei @tonychenxyz yup it does :) https://t.co/HZJSJeY1eM

[Sachin Konan @SachinKonan]

Big thanks to my advisor @liuzhuang1234 for the support and guidance! Joint work with @liujonathan24.

We're releasing our code, models, and datasets below:

Try it: https://t.co/7TDKkbGKlb
📝 Blog: https://t.co/QxJkwCGA9i
💻 Code: https://t.co/JJyrPhVN6O
📦 Models/data: https://t.co/KgxaEtPTrA
Paper: https://t.co/OrzcjlXHLH

[Sachin Konan @SachinKonan]

We also built tools to use PaperScorer on your own research!

You can:
- Run AI-assisted reviews + PaperScorer
- Score a LaTeX project
- Track how acceptance probability changes across Git commits as shown below

See whether your revisions are moving the needle 📈