We Need to Desk Reject More and Faster

We need to desk reject more. The continuous increase in submissions to top AI conferences is putting the peer reviewing system under a huge pressure. Although I don’t have the numbers, I believe AI slop is partially causing this along with other reasons. We can put a stop to it, but only if we stop treating obviously flawed papers with kid gloves. We need to desk reject, and we need to do it now.

In the last three conferences and workshops I have reviewed, out of a total of 12 submissions, 4 had clear hallucinations in the references or breaks in the submission guidelines. This is just anecdotical data, but I have heard several friends in similar situations. Given that we don’t have enough reviewers, we don’t have the luxury to allow these papers to clog the reviewing system.

In most conferences, reviewers expect submissions to be in good faith, so we review them and, after spending hours, we eventually find these problems. We need to change this system to something similar to ACL, where during the first two days of the reviewing period, reviewers check whether papers deserve a full review. However, ACL is not taking this step seriously enough. I have identified clear desk reject reasons for two papers, and after five days I still have no answer. Other papers are waiting for reviewers; I could be reviewing one of them if we desk rejected the not-in-good-faith submissions in my batch.

Given the difficulty of finding reviewers for AI conferences, we need to be stricter with what we accept for a full review. Submissions with hallucinated references, and more generally, those breaking the submission guidelines, should be desk rejected fast so that these reviewers can focus on submissions in good faith. After all, if authors don’t put in the effort to clean errors from LLM outputs, why should reviewers do that?