[TYPES] Two-tier reviewing process

[Stephan Zdancewic]

Following up on Benjamin's comments about "X" reviews.  There are two 
different axes that are important when understanding a review.  One is 
the reviewer's  *expertise* in the subject of the paper.  Another is the 
reviewer's *confidence* in his or her assessment carried out in the 
review.  Using only one score to indicate both leads to some confusion, 
since the two properties get conflated.  As Benjamin suggests, I often 
find myself wanting to indicate confidence when I'm not an expert, and 
sometimes, even though I'm an expert in terms of the related work, I 
still don't have high confidence in my review (perhaps because it's a 
really intricate paper).

--Steve


Benjamin Pierce wrote:
> [ The Types Forum, http://lists.seas.upenn.edu/mailman/listinfo/types-list ]
>
> I'm suspicious of statistics about number of X reviews.  I know what X  
> is supposed to mean ("I am an expert in the topic") but in practice I  
> see many people (including myself) acting as if it means "I understood  
> the paper completely," and therefore often falling back to Y to  
> indicate things like "Although I'm an expert, the paper was poorly  
> explained and I couldn't completely understand it in a reasonable  
> amount of time."
>
> This isn't to say that comparing figures for POPL and ICFP is not  
> worthwhile -- just that the numbers themselves should be taken with a  
> grain of salt.
>
>      - Benjamin
>
>
> On Jan 27, 2010, at 9:11 PM, Norman Ramsey wrote:
>
>   
>> [ The Types Forum, http://lists.seas.upenn.edu/mailman/listinfo/types-list 
>>  ]
>>
>>     
>>> At the POPL discussion, one goal that was raised was to improve the
>>> number of "expert" reviews per paper. People are dissatisfied when
>>> their paper is rejected by self-proclaimed non-experts.  I believe
>>> that Jens pointed out that this year's POPL had 77% papers with one
>>> "X" review.
>>>       
>> I went back and got archival data for ICFP 2007.  ICFP is a
>> significantly smaller conference which that year had only 120
>> submissions.  110 of 120 submissions (91%) received at least one X
>> review.  When comparing these data, here are some points to keep in
>> mind:
>>
>>  - ICFP reviewing was double-blind that year.
>>  - Otherwise ICFP used substantially the same review process that
>>    POPL uses now.
>>  - POPL is probably a broader conference than ICFP, which may make it
>>    more difficult to find expert external reviewers.
>>
>> I remember great difficulty in finding external reviewers for papers
>> involving functional programming and XML---many were multi-author
>> papers, and this is a small community with a lot of cross-
>> fertilization, so there were quite a few papers for which all the
>> obvious expert reviewers had conflicts.  (One of the problems with
>> double-blind review is that it makes a prudent program chair more
>> cautious about conflicts of interest.)
>>
>>
>>
>> Norman
>>     
>
>