[TYPES] Two-tier reviewing process

[Benjamin Pierce]

I'm suspicious of statistics about number of X reviews.  I know what X  
is supposed to mean ("I am an expert in the topic") but in practice I  
see many people (including myself) acting as if it means "I understood  
the paper completely," and therefore often falling back to Y to  
indicate things like "Although I'm an expert, the paper was poorly  
explained and I couldn't completely understand it in a reasonable  
amount of time."

This isn't to say that comparing figures for POPL and ICFP is not  
worthwhile -- just that the numbers themselves should be taken with a  
grain of salt.

     - Benjamin


On Jan 27, 2010, at 9:11 PM, Norman Ramsey wrote:

> [ The Types Forum, http://lists.seas.upenn.edu/mailman/listinfo/types-list 
>  ]
>
>> At the POPL discussion, one goal that was raised was to improve the
>> number of "expert" reviews per paper. People are dissatisfied when
>> their paper is rejected by self-proclaimed non-experts.  I believe
>> that Jens pointed out that this year's POPL had 77% papers with one
>> "X" review.
>
> I went back and got archival data for ICFP 2007.  ICFP is a
> significantly smaller conference which that year had only 120
> submissions.  110 of 120 submissions (91%) received at least one X
> review.  When comparing these data, here are some points to keep in
> mind:
>
>  - ICFP reviewing was double-blind that year.
>  - Otherwise ICFP used substantially the same review process that
>    POPL uses now.
>  - POPL is probably a broader conference than ICFP, which may make it
>    more difficult to find expert external reviewers.
>
> I remember great difficulty in finding external reviewers for papers
> involving functional programming and XML---many were multi-author
> papers, and this is a small community with a lot of cross-
> fertilization, so there were quite a few papers for which all the
> obvious expert reviewers had conflicts.  (One of the problems with
> double-blind review is that it makes a prudent program chair more
> cautious about conflicts of interest.)
>
>
>
> Norman