[TYPES/announce] Two phase reviewing for POPL; a response

[Derek Dreyer]

>> Anyone who's sat in a pc meeting knows there will be papers
>> where expert reviews are lacking, and someone volunteers to
>> read the paper overnight and provide an opinion.
>
> -- I think this could be solved by requiring reviews be due
> 1 week (or 2 weeks) before the meeting.  At that point,
> the PC could identify those papers with low quality reviews and
> solicit another one or two from experts.  This minor deviation
> from current practice would seem to be a direct solution to
> the stated problem.

In fact, this *is* the current practice, at least for POPL.  In the
past few years, the author response period has occurred a full week
before the PC meeting, and I know for a fact that a number of expert
reviews have been requested during that period.

Moreover, for POPL 2011, the PC is planning an even longer period of 2
weeks between the author response period and the physical PC meeting,
precisely in order to allow sufficient time to identify and debate
contentious and/or inadequately-reviewed papers online, and to seek
further expert reviews for them.

Regarding all of Dave's other comments, I agree with them 100%.

Derek

> -- Another solution:  Don't have double-blind reviewing.  Double-blind
> reviewing requires going through the PC chair to get external reviews,
> which disincentivizes getting these reviews even when you aren't an
> expert yourself.
>
> -- Another solution:  Expand the PC and/or have a greater percentage of the
> reviews be from external reviewers.  The goal would be to reduce the number
> of papers each PC member reviews.  My current experience with PLDI suggests
> I have a fixed total amount of time & energy for reviewing regardless of how
> many papers I have to review.  If I had to review 15-20 papers instead of 25,
> my 15-20 reviews would be better than the 25.  I also wonder, if there
> was 2-phase reviewing, whether my first phase reviews may be worse, because I have
> to save time and energy for the second phase.  Or alternatively, I wonder if
> might be too burned out by the first phase to do anything useful for the
> second phase.
>
>> They will
>> also know that innovative papers often have flaws, and the
>> dynamics of pc meetings is that the detractors usually win
>> out over the champions.  These are the issues that two-phase
>> reviewing is intended to address.
>
> -- Will more rounds of reviewing or more in-depth reviewing flip
> the dynamic so that champions win over detractors?  If somebody
> wants to trash a paper, they can.  With more effort invested,
> they can generally find more holes to poke in the work.  I'm
> skeptical the proposed scheme will help solve this problem.
>
> --------------
>
> Two other comments:
>
> -- I would massively disagree with any proposal in which some papers are not
> presented or there is disproportionate talk time given to certain papers,
> especially based on voting.  Who would ever vote to have a student you have
> never heard of give a talk over Simon Peyton Jones?  No one. I'd be voting for
> Simon every single time without even bothering to look at the paper!  And imagine
> your student has one great result and one great POPL paper and a coin flip
> means they don't get to present.  I could easily see this leading to the
> student missing out on getting interviews.
>
> -- If you want to change journal/conference culture, let's not change POPL or
> PLDI.  Let's change TOPLAS instead:  have TOPLAS papers accepted that year
> presented at POPL or PLDI or ICFP or OOPSLAA to attract attention to them.
>
> Dave
>