[TYPES/announce] Two phase reviewing for POPL; a response

[Robert Harper]

I am in complete agreement with Simon, and strongly opposed to the  
proposed plan to change the POPL reviewing process.

Bob Harper

On Jan 11, 2010, at 5:46 PM, Simon Peyton-Jones wrote:

> [ The Types Forum (announcements only),
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>
> Colleagues
>
> | Request for comments: Two-phase reviewing for POPL
> ...
> | The POPL Steering Committee has formulated the following proposal,
> | which we are circulating for discussion and feedback from the
> | community.  The proposal aims to improve the decision process for  
> POPL
> | while still working in a fixed time frame and with bounded  
> resources.
>
> Thank you for broadcasting the proposal, and offering the opportunity
> for feedback. I can't come to POPL this year, but I do have opinions
> about this proposal, so I thought I would put them in writing.  I'm
> sending this response only to the TYPES mailing list.
>
> Many people are concerned about the publication norms that have
> developed in our field [1,2,3,4].  In particular, we have evolved a
> somewhat bizarre system in which we place tremendous weight on
> publication in premier conferences with extremely low acceptance
> rates.  Promotion and tenure can depend on publication in these
> venues.  Yet anyone who has served on a program committee knows that
> (a) the evaluation is fairly rough and ready, and (b) it is hard to
> avoid a tendency to pick well-executed but incremental papers over
> more adventurous but flawed work.
>
> The current proposal for POPL is presumably a direct response to this
> situation. But I believe its main thrust, to invest yet more effort in
> the selection process, is addressing the wrong problem.  The problem  
> is
> not that program committees are selecting the *wrong* papers.  The
> problem is that they are selecting too *few* papers.
>
> Before developing these claims, I want to mention some real  
> advantages of
> the current conference system.
>
> * It is quick -- and *predictably* quick. There is a delay of only a
>  few months between submission and presentation; and there is never
>  any slippage, because the conference itself is immoveable.
>
> * It is a *fantastic* deal for authors. The most precious commodity  
> for
>  any author is the focused attention of other experts in the field.
>  When I began my academic career an author would be lucky to get
>  three scrawled sentences of review, on physical scraps of paper.
>  Nowadays authors get between three and six substantial, thoughtful
>  reviews.  That is gold dust.
>
> * Reviewing is recognised to be rough and ready.  Everyone knows that
>  there is no time to hunt for the perfect reviewer. The reviewers
>  know they have limited time for their work, and cut their cloth
>  accordingly.  For that very reason they are more inclined to agree
>  to write a review than if they are asked to review a 60-page journal
>  paper when they are supposed to do a bang-up thorough job.  Program
>  committee members review 20-30 papers, and simply cannot spend days
>  on each; and the universal acceptance of this fact is what makes
>  people willing to serve on PCs
>
>  I regard this limited time-budget for each review as a major
>  advantage.  80% of the benefit of a review comes from the first 20%
>  of investment.  Yes, individual injustices are sometimes done, and
>  all of us have been on the receiving end, but in the aggregate it is
>  a very efficient evaluation mechanism.  That is, it is not
>  perfectly accurate, but it is a *very effective use of reviewing
>  bandwidth*.
>
> * Much has been written about the evils of banging out papers to meet
>  conference deadlines, and no one would defend salami-slicing
>  incremental papers instead of working in a sustained way on
>  adventurous research.
>
>  Less has been written about the intellectual *advantages* of writing
>  frequently. My own experience is that the act of writing a paper is
>  tremendously enlightening.  I learn that I do not understand what I
>  though I understood.  The act of putting ideas onto paper forces
>  clarity, or at least exposes muddy thinking.  It puts thoughts into
>  a form when they can be shared with others.
>
>  Since I am a weak mortal, the incentive of a conference deadline is
>  often just what I need to force me to action.
>
> In short, there are really good things about our current system that  
> we
> do not want to lose.
>
> All that said, clearly something is wrong at the moment.  POPL is
> getting 250 submissions, and accepting 30-40.  That means that many
> fine papers are being rejected, and among the best 60 papers there is
> a strong element of chance about which ones end up being accepted.
> The same is true of PLDI, and perhaps to a lesser extent, of ICFP.
> (I don't have personal experience of the OOPSLA program committee.)
>
> We cannot fix this, as some would wish, by changing the culture to  
> make
> journal publications be regarded as more valuable than conference
> ones.  If this happened, the spotlight would just shift to journals,
> which would be overwhelmed with submissions; and we would lose many
> of the advantages I outline above.  But in any case it's a
> non-starter. No one can wave such a magic wand: cultures are *hard* to
> shift.
>
> Nor can we fix the problem by investing more effort in the review
> process, as the POPL committee is apparently suggesting.  We are
> already investing quite enough!  I'm all for careful reviewing.
> Double-blind reviewing (if done with a light touch, so that it does
> not cramp the authors style), and the opportunity for authors to rebut
> factual errors in reviews, both seem to have a good power-to-weight
> ratio.  But adding a whole new round of reviewing would represent an
> enormous new investment on the part of both authors and reviewer, and
> to what end?  Perhaps the published papers would be a little bit
> better, and the decisions would be a little bit more just.  But the
> costs are heavy, the benefits are marginal, and it addresses none of
> the fundamental problems.  I for one would think three times about
> agreeing to serve on such a PC.  (I already think twice.)
>
>
> No, the trouble is that POPL and conferences like it simply rejects
> too many fine, publishable papers.  This is bad because
>
>  - Authors are badly served, obviously
>
>  - Readers are badly served, because they don't get to read
>    those papers
>
>  - The papers get recycled at other conferences and workshops, where
>    they increase reviewing load (by being reviewed a second time),
>    and crowd out the truly workshop-y work in progress that should be
>    showing up at workshops
>
> In short, we should just accept more papers at all our premier
> conferences, using a *quality* bar (is this paper good enough?) not a
> *quantity* bar (is it one of the best 30?).  How can we do this?  The
> "fat proceedings" problem is getting less and less important as we
> increasingly use digital media.  Really the only difficulty is how to
> accommodate the presentations at the physical meeting itself. But this
> is a problem that could be dealt with in many ways.
>
> One straightforward one is to have parallel tracks.  Another is to
> have more days.  Still another, which I am rather fond of, is to
> accept (say) 60 papers, and then hold a lottery for 20 presentation
> slots.  [That's fewer than usual, so there'd be longer breaks for
> mingling, which is actually the real reason most people go to
> conferences in the first place.]  I would argue *against* choosing the
> "best" papers for presentation, because that will just re-introduce
> the ills we are currently struggling with.  Make it clearly a matter
> of luck, then no one will read anything into the "chosen for
> presentation" badge.
>
> The lottery selection could even done at the conference itself.  I'm
> only half joking; that way, no one could be denied travel funding on
> the grounds that his or her paper had not been chosen for
> presentation.  Or perhaps participants registering for the conference
> could vote in advance for which accepted papers they'd like to see
> presented, so the programme is partly created by those attending the
> conference?
>
>
> A big advantage of this approach (simply accepting more papers) is
> that it is something we can simply choose to do.  It does not require
> every conference to make the same choice simultaneously, and it
> doesn't require a magical cultural change.  However, if we did take
> this path, then a significant cultural change would follow, over time.
> If publication at POPL was no longer an extraordinary achievement, but
> rather a recognition for a fine piece of work, appointment committees
> would in due course adjust their evaluation criteria.  And that in
> turn might actually reduce the overwhelming number of submissions to
> top-drawer conferences.
> Simon Peyton Jones
>
> [1] J Wing, "CS woes: deadline-driven research, academic
>     inequality", CACM 52(12) Dec 2009, p8
>
> [2] J Crowcroft, S Keshav and N McKeown, "Scaling the academic
>    publication process to internet scale", CACM 52(1), Jan 2009,
>    pp27-30.
>
> [3] M Vardi, "Conferences vs journals", CACM 52(5), May 2009, p5
>
> [4] K Bierman, FB Schneider "Program committee overload in systems",
>    CACM 52(5), May 2009
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2108 bytes
Desc: not available
Url : http://lists.seas.upenn.edu/pipermail/types-announce/attachments/20100111/6d995185/smime-0001.p7s