[TYPES/announce] Two phase reviewing for POPL; a response

[Mitchell Wand]

"lek"-- what a wonderful word!  Thank you, Prakash, for bringing it to our
attention.  Now on to more serious comments:

I agree with most of the commenters here that the POPL proposal goes in the
wrong direction.  The fact that conference acceptances is so stochastic
leads to a variety of bad outcomes:

1.  People submit more papers.   Some of these papers are recycled
rejections looking for a different PC.   Sometimes the comments of the first
PC improve the paper for the second go-round, but my impression is that this
is the exception.   I suspect that the low acceptance rates lead to finer
bologna-slicing,  though I don't have any data to back this up.  NSF
observed this phenomenon when their acceptance rates got too low.   (Their
solution was draconian:  only have a call for proposals every other year,
and restrict PIs to one submission per competition.  Not desirable.)

2. There is inversion of quality.  I recall when PPDP was co-located with
ICFP that some people would send their better papers to PPDP rather than
risk rejection at ICFP.

3. There are second-order effects:  For example, we require our PhD students
to get an accepted paper as part of our qualifying process.  We've seen
perfectly good papers rejected multiple times, and we've had to create a
whole structure to work around this bug.   Predictable conference
acceptances would avoid this problem.   People on this thread have already
mentioned this problem with respect to hiring.   For tenure and promotion,
it is probably diminished due to the longer time frame, but I'd bet it's
still there.

I believe that we need to get our acceptance ratio up to about 40%.   I
think this would drastically increase the predictability of the process.
>From my time on POPL and ICFP PC's I suspect that we could reach this goal
without substantial loss of quality.

1.  I now think that having parallel tracks is a good idea.  I used to think
otherwise, but I have found it exciting to bounce around, as we do on
workshop days, from one workshop to another.  It also leads me to interact
with more sub-communities than I would otherwise.

2. I remain dubious about poster sessions.   These make it difficult for
presenters to see others' presentations.  It's also harder to tune out when
a presentation turns out not to be of interest-- it's hard to walk away.

3. I would support a system of multiple durations, in which the PC would
assign different-length time slots to different papers based on perceived
quality of work, likely interest in the presentation, etc.  If we adopt such
a system, however, it is important that all accepted papers receive the same
allocation of pages in the printed proceedings.

4. Getting good reviews is a problem, but this is a separate problem and
should be dealt with separately.  I suspect that the quality of the reviews
is less of a problem at a 40% acceptance ratio than it is at 20%: a
poor-quality review is less likely to sway the outcome.

What would happen if we announced that we were doubling the number of
accepted papers to POPL?   In the first year, at least, I suspect that we
would get many more submissions than in the past. (I'd therefore publicly
state the goal in terms of number of acceptances, not in terms of acceptance
ratio).  Other conferences, eg ICFP and PLDI, might be forced to follow suit
in order to keep their submission levels up.   So we might be initiating a
virtuous cycle, in which the number of good papers and the number of slots
in conferences were more in balance.

The additional submissions would be something of a burden to the PC
committee, but probably many of the "extra" submissions would be of poor
quality, and would not require much effort.

I hope these thoughts are useful.

--Mitch


On Tue, Jan 12, 2010 at 9:18 AM, Prakash Panangaden <prakash at cs.mcgill.ca>wrote:

> [ The Types Forum (announcements only),
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>
> Thanks to Simon Peyton-Jones for sharing his articulate and
> well-reasoned thoughts with us.  I agree that there are too few papers
> accepted at the best conferences and that the problem is not quality of
> reviewing.  I would like to push for the idea that we accept many more
> papers (perhaps double the present number) and present them at poster
> sessions and have them appear in the proceedings as is done at most of
> the big AI conferences.  Then we could have a smaller number presented
> as conference presentations.  The other point I would like to make is
> that we as reviewers are far too obsessed with polished but incremental
> papers and in the theory conferences (STOC/FOCS/LICS/ICALP) with "hard
> but boring" problems.  It is indeed hard to change the culture, but
> conferences are where we should get the chance to throw out ideas rather
> than participate in a lek.
> Cheers
> Prakash
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.seas.upenn.edu/pipermail/types-announce/attachments/20100112/611d88d0/attachment-0001.htm