[TYPES/announce] Two phase reviewing for POPL; a response

[Simon Peyton-Jones]

Colleagues

| Request for comments: Two-phase reviewing for POPL
...
| The POPL Steering Committee has formulated the following proposal,
| which we are circulating for discussion and feedback from the
| community.  The proposal aims to improve the decision process for POPL
| while still working in a fixed time frame and with bounded resources.

Thank you for broadcasting the proposal, and offering the opportunity
for feedback. I can't come to POPL this year, but I do have opinions
about this proposal, so I thought I would put them in writing.  I'm
sending this response only to the TYPES mailing list.

Many people are concerned about the publication norms that have
developed in our field [1,2,3,4].  In particular, we have evolved a
somewhat bizarre system in which we place tremendous weight on
publication in premier conferences with extremely low acceptance
rates.  Promotion and tenure can depend on publication in these
venues.  Yet anyone who has served on a program committee knows that
(a) the evaluation is fairly rough and ready, and (b) it is hard to
avoid a tendency to pick well-executed but incremental papers over
more adventurous but flawed work.

The current proposal for POPL is presumably a direct response to this
situation. But I believe its main thrust, to invest yet more effort in
the selection process, is addressing the wrong problem.  The problem is
not that program committees are selecting the *wrong* papers.  The
problem is that they are selecting too *few* papers.

Before developing these claims, I want to mention some real advantages of
the current conference system.

* It is quick -- and *predictably* quick. There is a delay of only a
  few months between submission and presentation; and there is never
  any slippage, because the conference itself is immoveable.

* It is a *fantastic* deal for authors. The most precious commodity for
  any author is the focused attention of other experts in the field.
  When I began my academic career an author would be lucky to get
  three scrawled sentences of review, on physical scraps of paper.
  Nowadays authors get between three and six substantial, thoughtful
  reviews.  That is gold dust.

* Reviewing is recognised to be rough and ready.  Everyone knows that
  there is no time to hunt for the perfect reviewer. The reviewers
  know they have limited time for their work, and cut their cloth
  accordingly.  For that very reason they are more inclined to agree
  to write a review than if they are asked to review a 60-page journal
  paper when they are supposed to do a bang-up thorough job.  Program
  committee members review 20-30 papers, and simply cannot spend days
  on each; and the universal acceptance of this fact is what makes
  people willing to serve on PCs

  I regard this limited time-budget for each review as a major
  advantage.  80% of the benefit of a review comes from the first 20%
  of investment.  Yes, individual injustices are sometimes done, and
  all of us have been on the receiving end, but in the aggregate it is
  a very efficient evaluation mechanism.  That is, it is not
  perfectly accurate, but it is a *very effective use of reviewing
  bandwidth*.

* Much has been written about the evils of banging out papers to meet
  conference deadlines, and no one would defend salami-slicing
  incremental papers instead of working in a sustained way on
  adventurous research.

  Less has been written about the intellectual *advantages* of writing
  frequently. My own experience is that the act of writing a paper is
  tremendously enlightening.  I learn that I do not understand what I
  though I understood.  The act of putting ideas onto paper forces
  clarity, or at least exposes muddy thinking.  It puts thoughts into
  a form when they can be shared with others.

  Since I am a weak mortal, the incentive of a conference deadline is
  often just what I need to force me to action.  

In short, there are really good things about our current system that we 
do not want to lose.

All that said, clearly something is wrong at the moment.  POPL is
getting 250 submissions, and accepting 30-40.  That means that many
fine papers are being rejected, and among the best 60 papers there is
a strong element of chance about which ones end up being accepted.  
The same is true of PLDI, and perhaps to a lesser extent, of ICFP.
(I don't have personal experience of the OOPSLA program committee.)

We cannot fix this, as some would wish, by changing the culture to make
journal publications be regarded as more valuable than conference
ones.  If this happened, the spotlight would just shift to journals,
which would be overwhelmed with submissions; and we would lose many
of the advantages I outline above.  But in any case it's a
non-starter. No one can wave such a magic wand: cultures are *hard* to
shift.

Nor can we fix the problem by investing more effort in the review
process, as the POPL committee is apparently suggesting.  We are
already investing quite enough!  I'm all for careful reviewing.
Double-blind reviewing (if done with a light touch, so that it does
not cramp the authors style), and the opportunity for authors to rebut
factual errors in reviews, both seem to have a good power-to-weight
ratio.  But adding a whole new round of reviewing would represent an
enormous new investment on the part of both authors and reviewer, and
to what end?  Perhaps the published papers would be a little bit
better, and the decisions would be a little bit more just.  But the
costs are heavy, the benefits are marginal, and it addresses none of
the fundamental problems.  I for one would think three times about
agreeing to serve on such a PC.  (I already think twice.)


No, the trouble is that POPL and conferences like it simply rejects
too many fine, publishable papers.  This is bad because

  - Authors are badly served, obviously

  - Readers are badly served, because they don't get to read 
    those papers

  - The papers get recycled at other conferences and workshops, where
    they increase reviewing load (by being reviewed a second time),
    and crowd out the truly workshop-y work in progress that should be
    showing up at workshops

In short, we should just accept more papers at all our premier
conferences, using a *quality* bar (is this paper good enough?) not a
*quantity* bar (is it one of the best 30?).  How can we do this?  The
"fat proceedings" problem is getting less and less important as we
increasingly use digital media.  Really the only difficulty is how to
accommodate the presentations at the physical meeting itself. But this
is a problem that could be dealt with in many ways.

One straightforward one is to have parallel tracks.  Another is to
have more days.  Still another, which I am rather fond of, is to
accept (say) 60 papers, and then hold a lottery for 20 presentation
slots.  [That's fewer than usual, so there'd be longer breaks for
mingling, which is actually the real reason most people go to
conferences in the first place.]  I would argue *against* choosing the
"best" papers for presentation, because that will just re-introduce
the ills we are currently struggling with.  Make it clearly a matter
of luck, then no one will read anything into the "chosen for
presentation" badge.

The lottery selection could even done at the conference itself.  I'm
only half joking; that way, no one could be denied travel funding on
the grounds that his or her paper had not been chosen for
presentation.  Or perhaps participants registering for the conference
could vote in advance for which accepted papers they'd like to see
presented, so the programme is partly created by those attending the
conference?


A big advantage of this approach (simply accepting more papers) is
that it is something we can simply choose to do.  It does not require
every conference to make the same choice simultaneously, and it
doesn't require a magical cultural change.  However, if we did take
this path, then a significant cultural change would follow, over time.
If publication at POPL was no longer an extraordinary achievement, but
rather a recognition for a fine piece of work, appointment committees
would in due course adjust their evaluation criteria.  And that in
turn might actually reduce the overwhelming number of submissions to
top-drawer conferences.
Simon Peyton Jones

[1] J Wing, "CS woes: deadline-driven research, academic
     inequality", CACM 52(12) Dec 2009, p8 

[2] J Crowcroft, S Keshav and N McKeown, "Scaling the academic
    publication process to internet scale", CACM 52(1), Jan 2009,
    pp27-30.

[3] M Vardi, "Conferences vs journals", CACM 52(5), May 2009, p5

[4] K Bierman, FB Schneider "Program committee overload in systems",
    CACM 52(5), May 2009