[TYPES/announce] Two phase reviewing for POPL; a response

[Derek Dreyer]

Hi, Matthias.

Thanks for the clarification about the motivation behind the rather
significant proposed change to the POPL review process.

There seem to be three major concerns: (1) rejecting papers without at
least one expert review, (2) rejecting papers just because of a
(presumably fixable) weakness, and (3) rejecting papers because they
are "quirky".  Let's consider each in turn.

1. I completely understand the concern about papers receiving no
expert reviews.  This has happened to everyone on occasion, I'm sure.
However, at least based on my (perhaps confused) understanding of the
proposal at hand, I fail to see how it actually addresses that
concern.

Let's say I submit a paper to POPL and it gets no expert reviews.  The
PC discovers this at the PC meeting, and thus they can't make a
serious decision and they return my paper "resubmit".  Since I have
not received an expert review, I get little serious feedback at this
point, so I can't make any serious changes to my paper that would help
reviewers understand it.  In the meantime, the PC seeks out some
external reviews.

What is the point of this protracted process?  The problem in this
case is that the PC goofed up by not procuring an external expert
review in time for the PC meeting.  Why is the poor author punished by
having to wait an extra 6 weeks to get an acceptance or rejection?

A simpler solution, it seems to me, is for the PC chair (and if there
are co-chairs, this will hopefully be easier) to make sure there are
expert reviews for all papers, including at least one external expert
review, preferably prior to the author response.

2. Regarding papers with a fixable weakness, I tend to agree with
Sophia.  Why is this not already handled by shepherding?  Although I
am not a big fan of shepherding, I think it is appropriate to use in
the case that a paper contains clear technical flaws that seem easily
fixable.  Are there really that many papers that fall into this
category that it's worth making such a big change to the POPL
reviewing process?

3. Regarding "quirky" papers: I have always interpreted "quirky" to
mean that some people think the paper is "really cool" and others
think it's "not there yet" and/or "rubbish".  Again, I agree with
Sophia: I'm not sure how the new reviewing process is supposed to help
make decisions about such papers.  Generally, I think, with these
sorts of papers, it's not necessarily the presentation of the work
that's at fault; in fact, the offbeat presentation may be why some
people like it.  Rather, it's a question of whether the work is solid
enough for POPL, and that's unlikely to be something that can be
addressed in a 2-week revision period.

***

To echo Simon and the others, I'm happy to see the POPL acceptance
rate go up to 25-30%.  If the policy is that we accept more good
papers up to that acceptance rate, then I don't see what is difficult
about implementing it.  It is a simple fix, and does not affect the
conference calendar.

I am not in favor of a lottery for talks, but parallel sessions seem
like a reasonable price to pay.

Best regards,
Derek

On Tue, Jan 12, 2010 at 4:23 AM, Matthias Felleisen
<matthias at ccs.neu.edu> wrote:
> [ The Types Forum (announcements only),
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>
>
> Simon and others,
>
> thanks for the feedback. While I am a member of the POPL SC until
> January, this email is my personal opinion and is not to be
> construed as something anyone else on the SC may subscribe to.
>
>
> 1. I like the idea of accepting more papers. It is a part of
> every year's SC discussion. And as you probably recall from your
> chairman role, the SC is quite encouraging about accepting
> more papers. I tried to get more papers into my POPL. This
> year again the SC is encouraging the PC to broaden the
> acceptance range.
>
> In my own experience, though, the PC tends to block attempts
> to, say, double the number of papers. The idea is discussed,
> potential papers are brought up, and eventually the faction
> that disapproves of a larger program wears out those who are
> on the 'more papers' side.
>
> If you wish to achieve this numeric enlargement, I am afraid
> you will need to set a target number, say 60 papers.
>
> 2. I strongly disagree with your description of the state of
> affairs:
>
>> When I began my academic career an author would be lucky to get
>>  three scrawled sentences of review, on physical scraps of paper.
>>  Nowadays authors get between three and six substantial, thoughtful
>>  reviews.  That is gold dust.
>
> I started with the same kind of paper reviews. I do not see
> much of the improvement you see.
>
> On some occasion, you get fantastic reviews (even for rejected
> papers).
>
> On many occasions, you get feedback from non-experts that is
> nearly random. I won't blame the committee members here: they
> are under time pressure, they have their own preferences,
> and they have the backgrounds they have. We are all weak, and
> we do what we can do.
>
> Over the past few years, I have collected informal statistics
> that basically suggests that submissions get about one expert
> review per submission but that may mean two for one submission
> and none for another. Yes, a good number of papers are rejected
> without knowledgeable review.
>
> The expectation with a two-phase review is this:
>
> -- At the first, physical meeting papers w/o expert reviews
> are discovered. They get to respond to their preliminary reviews
> and the PC will figure out how to get appropriate reviewers
> for these papers. No more 'over night, after dinner'
> expert reviews.
>
> -- Papers with a fixable weakness are sent back to authors so
> that they can demonstrate clearly how to overcome this weakness.
> Hopefully this will help get some less-finished ideas
> into the conference and turn the 'oral journal' that POPL currently
> is into a real conference again.
>
> 3. The two-tier proposal is a compromise that a dozen or so
> representative POPL community members have worked out. Like all
> compromises they don't make people 100% happy. I can only speak
> for myself; I am in the not-100% happy camp.
>
> But after discussing this issue for the entire fall, I believe that
> the (once again) suggested increase of acceptances and this
> proposal may get POPL moving in the right direction. I do consider it
> an experiment -- as I am sure others on the SC do, too -- that should
> be evaluated in a few years. If it moves in the wrong direction,
> we go back to the status quo or we try a different change.
> It it turns out to be an improvement, we can still try other
> modifications.
>
> -- Matthias
>
>
>
>
>
>
>
>
>