[TYPES/announce] Two phase reviewing for POPL; a response

[Michael Hicks]

It seems like a common desire in the many interesting responses so far  
is to increase the effectiveness of the review process, to accept  
deserving papers, and to provide good feedback to papers that need a  
bit more work.

I'd just like to add my voice to Andrew's (below) that tiered  
reviewing works well toward achieving these goals, somewhat to my  
surprise.  The multiple phases of review increases the chances of  
obtaining expert reviews.  The filtering in the early phases ensures  
that interesting, but not universally-acceptable papers get more  
reviews, which improves the quality of feedback.  I was a bit worried  
that early-phase filtering might unfairly remove papers from the  
process, but I don't believe that has ever happened on the PCs I've  
been on; i.e., even a fairly conservative criterion for eliminating  
papers from further consideration (e.g., the first two reviews are  
both 'D') worked very well.  I'm sure Andrew can provide more stats on  
this, for those interested.

Also, I've found the PLDI and ISMM "review committee" approach to  
expanding the review pool to work pretty well, but I have no general  
stats on that.

-Mike

On Jan 12, 2010, at 9:57 AM, Andrew Myers wrote:

> [ The Types Forum (announcements only),
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>
> The current two-phase proposal sounds to me as if it will  
> significantly
> increase the amount of reviewing work without significantly increasing
> the quality of the reviewing process, for the reasons Derek has  
> argued.
>
> Why not look at the approaches other communities have taken? Several
> major conferences in the networking, systems and security communities
> have changed to a different two- or three-phase reviewing process in
> which papers are rejected early in the process if they have enough
> confident negative reviews. Only the best papers and papers with low
> confidence continue on. I've seen this both as a PC member and as a PC
> chair, and in my experience, it's great. The reviewing load is  
> increased
> only slightly, and both the quality of the reviews and the quality of
> the decision process is improved. It's also more fun and educational  
> to
> be a PC member, because the average quality of the papers you review  
> is
> higher. And the accepted papers get more reviews, which also improves
> the product.
>
> I believe Tom Anderson first introduced this idea for SIGCOMM 2006; I
> have more detailed notes on how this worked for IEEE Security and
> Privacy (Oakland) 2009.
>
> Cheers,
>
> -- Andrew