https://www.sciencedaily.com/releases/2023/08/230801152846.htm

In many review programs, say the authors, proposals are funded primarily on the basis of reviewer ratings and summary statistics derived from them that represent reviewers' assessment of proposal quality. Relying solely on ratings, however, can pose problems for establishing funding priorities, "especially when distinguishing between similarly rated proposals, especially for projects of high quality, where only a few minor weaknesses can make the difference between a funded project and a rejection." Furthermore, say Gallo and colleagues, the use of ratings rubrics "often forces reviewers to penalize a project based on a series of identified but minor weaknesses, despite the fact that they may find the application overall to be potentially significant."

To address these concerns, the authors suggest adding rankings to the traditional rating approach and describe an innovative methodology called the Mallows-Binomial, which can then produce integrated proposal scores, as well as an induced preference ordering. In so doing, the approach allows funders to consider reviewers' scores alongside their opinions about how well the proposals stack up against one another. Pearce states that the inclusion of ranking will "more accurately distill reviewer opinion into a useful output to make an informed funding decision."