[TYPES/announce] Two phase reviewing for POPL; a response

[Andrew Myers]

Michael Hicks wrote:
> I'd just like to add my voice to Andrew's (below) that tiered  
> reviewing works well toward achieving these goals, somewhat to my  
> surprise.  The multiple phases of review increases the chances of  
> obtaining expert reviews.  The filtering in the early phases ensures  
> that interesting, but not universally-acceptable papers get more  
> reviews, which improves the quality of feedback.  I was a bit worried  
> that early-phase filtering might unfairly remove papers from the  
> process, but I don't believe that has ever happened on the PCs I've  
> been on; i.e., even a fairly conservative criterion for eliminating  
> papers from further consideration (e.g., the first two reviews are  
> both 'D') worked very well.  I'm sure Andrew can provide more stats on  
> this, for those interested.
>   
I've posted the part of the Oakland 2009 chair's report that talks about 
the three-round reviewing process used there to:

http://www.cs.cornell.edu/andru/oakland09/reviewing-slides.pdf

We had 253 submissions and got that down to 26 accepts with 5-8 reviews 
per accepted paper. Reviewing load was about 20-23 papers per PC member. 
There were 25 people at the physical PC meeting (I think a physical PC 
meeting is important for a 'top' conference in its area). The keys to 
making this work were 1) to have the PC split into 'heavy' and 'light' 
contingents -- this is different from the external review committee 
approach used in recent PL conferences, and 2) to filter papers in early 
rounds. To aid filtering, I think it was useful to have a 6-point rating 
scale with clearly defined semantics and ratings above A and below D 
(strong accept/reject).

The slides have the timeline and the numbers, which do matter if you're 
trying to engineer a similar process.

-- Andrew