[TYPES/announce] Two phase reviewing for POPL; a response

[David Walker]

> Anyone who's sat in a pc meeting knows there will be papers
> where expert reviews are lacking, and someone volunteers to
> read the paper overnight and provide an opinion.

-- I think this could be solved by requiring reviews be due
1 week (or 2 weeks) before the meeting.  At that point, 
the PC could identify those papers with low quality reviews and 
solicit another one or two from experts.  This minor deviation
from current practice would seem to be a direct solution to
the stated problem.  

-- Another solution:  Don't have double-blind reviewing.  Double-blind
reviewing requires going through the PC chair to get external reviews,
which disincentivizes getting these reviews even when you aren't an
expert yourself.

-- Another solution:  Expand the PC and/or have a greater percentage of the
reviews be from external reviewers.  The goal would be to reduce the number
of papers each PC member reviews.  My current experience with PLDI suggests 
I have a fixed total amount of time & energy for reviewing regardless of how 
many papers I have to review.  If I had to review 15-20 papers instead of 25, 
my 15-20 reviews would be better than the 25.  I also wonder, if there 
was 2-phase reviewing, whether my first phase reviews may be worse, because I have 
to save time and energy for the second phase.  Or alternatively, I wonder if
might be too burned out by the first phase to do anything useful for the 
second phase.  

> They will
> also know that innovative papers often have flaws, and the
> dynamics of pc meetings is that the detractors usually win
> out over the champions.  These are the issues that two-phase
> reviewing is intended to address.

-- Will more rounds of reviewing or more in-depth reviewing flip 
the dynamic so that champions win over detractors?  If somebody
wants to trash a paper, they can.  With more effort invested,
they can generally find more holes to poke in the work.  I'm
skeptical the proposed scheme will help solve this problem.  

--------------

Two other comments: 

-- I would massively disagree with any proposal in which some papers are not 
presented or there is disproportionate talk time given to certain papers, 
especially based on voting.  Who would ever vote to have a student you have 
never heard of give a talk over Simon Peyton Jones?  No one. I'd be voting for 
Simon every single time without even bothering to look at the paper!  And imagine
your student has one great result and one great POPL paper and a coin flip
means they don't get to present.  I could easily see this leading to the 
student missing out on getting interviews.  

-- If you want to change journal/conference culture, let's not change POPL or 
PLDI.  Let's change TOPLAS instead:  have TOPLAS papers accepted that year 
presented at POPL or PLDI or ICFP or OOPSLAA to attract attention to them.

Dave