[TYPES/announce] Two phase reviewing for POPL; a response

[Kim Bruce]

I also would like to agree with Simon's general thrust.  POPL and PLDI are so competitive that it is difficult for the program committee to take a chance on a speculative paper when it would squeeze out a clearly very good paper that is less speculative.  Thus I agree that we need to accept more papers at these conferences.

Serving on a program committee is very time-consuming.  If the POPL proposal were accepted as it stands, I would hope that only a few papers would be in the resubmit pile so that the workload is not overwhelming for the committee (and I agree that it is useful to have a method for authors to respond to referee comments that might reflect misunderstandings -- though a paper that was not well written could more appropriately be resubmitted to a later conference).

While I'm not a big fan of Simon's proposal of a lottery to decide which papers will be presented, I have fantasized over the years of having a mechanism where each speaker starts off by giving a 15 minute version of their talk.  The audience then votes to see if they continue for another 15 minutes.  It is completely unworkable, of course, but it would help in those cases that you decide to attend a talk and then decide after the first few minutes that it is either very different from what you expected or is not well presented.  Instead these days we simply flip open our laptops and catch up on our e-mail.

My choice would be to increase the acceptance rate and schedule all or some of the submitted papers in parallel sessions.  I know there are losses to this (we all become narrower -- we might also have less time to talk in the hallways if there are more papers that are personally interesting), but it will allow more high quality papers to be presented.  (Extending conferences is less useful as it is hard for many of us to be away for an extended time during classes.)  It also makes sense to do this if we as a profession are going to encourage more use of journals for archival publications.  

Conference program committees are forced to referee based on interest and likely correctness, while journals can have more detailed proofs, explanations, and data that can be verified by referees.  While, as Simon says, it is useful to write often about one's research, it is also useful to put together what one has learned in several successive conference papers to make an up-to-date and comprehensive report on one's research.


Kim Bruce

.. also not going to POPL this year ...

On Jan 11, 2010, at 2:46 PM, Simon Peyton-Jones wrote:

> [ The Types Forum (announcements only), 
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
> 
> Colleagues
> 
> | Request for comments: Two-phase reviewing for POPL
> ...
> | The POPL Steering Committee has formulated the following proposal,
> | which we are circulating for discussion and feedback from the
> | community.  The proposal aims to improve the decision process for POPL
> | while still working in a fixed time frame and with bounded resources.
> 
> Thank you for broadcasting the proposal, and offering the opportunity
> for feedback. I can't come to POPL this year, but I do have opinions
> about this proposal, so I thought I would put them in writing.  I'm
> sending this response only to the TYPES mailing list.
> 
> Many people are concerned about the publication norms that have
> developed in our field [1,2,3,4].  In particular, we have evolved a
> somewhat bizarre system in which we place tremendous weight on
> publication in premier conferences with extremely low acceptance
> rates.  Promotion and tenure can depend on publication in these
> venues.  Yet anyone who has served on a program committee knows that
> (a) the evaluation is fairly rough and ready, and (b) it is hard to
> avoid a tendency to pick well-executed but incremental papers over
> more adventurous but flawed work.
> 
> The current proposal for POPL is presumably a direct response to this
> situation. But I believe its main thrust, to invest yet more effort in
> the selection process, is addressing the wrong problem.  The problem is
> not that program committees are selecting the *wrong* papers.  The
> problem is that they are selecting too *few* papers.
> 
> Before developing these claims, I want to mention some real advantages of
> the current conference system.
> 
> * It is quick -- and *predictably* quick. There is a delay of only a
>  few months between submission and presentation; and there is never
>  any slippage, because the conference itself is immoveable.
> 
> * It is a *fantastic* deal for authors. The most precious commodity for
>  any author is the focused attention of other experts in the field.
>  When I began my academic career an author would be lucky to get
>  three scrawled sentences of review, on physical scraps of paper.
>  Nowadays authors get between three and six substantial, thoughtful
>  reviews.  That is gold dust.
> 
> * Reviewing is recognised to be rough and ready.  Everyone knows that
>  there is no time to hunt for the perfect reviewer. The reviewers
>  know they have limited time for their work, and cut their cloth
>  accordingly.  For that very reason they are more inclined to agree
>  to write a review than if they are asked to review a 60-page journal
>  paper when they are supposed to do a bang-up thorough job.  Program
>  committee members review 20-30 papers, and simply cannot spend days
>  on each; and the universal acceptance of this fact is what makes
>  people willing to serve on PCs
> 
>  I regard this limited time-budget for each review as a major
>  advantage.  80% of the benefit of a review comes from the first 20%
>  of investment.  Yes, individual injustices are sometimes done, and
>  all of us have been on the receiving end, but in the aggregate it is
>  a very efficient evaluation mechanism.  That is, it is not
>  perfectly accurate, but it is a *very effective use of reviewing
>  bandwidth*.
> 
> * Much has been written about the evils of banging out papers to meet
>  conference deadlines, and no one would defend salami-slicing
>  incremental papers instead of working in a sustained way on
>  adventurous research.
> 
>  Less has been written about the intellectual *advantages* of writing
>  frequently. My own experience is that the act of writing a paper is
>  tremendously enlightening.  I learn that I do not understand what I
>  though I understood.  The act of putting ideas onto paper forces
>  clarity, or at least exposes muddy thinking.  It puts thoughts into
>  a form when they can be shared with others.
> 
>  Since I am a weak mortal, the incentive of a conference deadline is
>  often just what I need to force me to action.  
> 
> In short, there are really good things about our current system that we 
> do not want to lose.
> 
> All that said, clearly something is wrong at the moment.  POPL is
> getting 250 submissions, and accepting 30-40.  That means that many
> fine papers are being rejected, and among the best 60 papers there is
> a strong element of chance about which ones end up being accepted.  
> The same is true of PLDI, and perhaps to a lesser extent, of ICFP.
> (I don't have personal experience of the OOPSLA program committee.)
> 
> We cannot fix this, as some would wish, by changing the culture to make
> journal publications be regarded as more valuable than conference
> ones.  If this happened, the spotlight would just shift to journals,
> which would be overwhelmed with submissions; and we would lose many
> of the advantages I outline above.  But in any case it's a
> non-starter. No one can wave such a magic wand: cultures are *hard* to
> shift.
> 
> Nor can we fix the problem by investing more effort in the review
> process, as the POPL committee is apparently suggesting.  We are
> already investing quite enough!  I'm all for careful reviewing.
> Double-blind reviewing (if done with a light touch, so that it does
> not cramp the authors style), and the opportunity for authors to rebut
> factual errors in reviews, both seem to have a good power-to-weight
> ratio.  But adding a whole new round of reviewing would represent an
> enormous new investment on the part of both authors and reviewer, and
> to what end?  Perhaps the published papers would be a little bit
> better, and the decisions would be a little bit more just.  But the
> costs are heavy, the benefits are marginal, and it addresses none of
> the fundamental problems.  I for one would think three times about
> agreeing to serve on such a PC.  (I already think twice.)
> 
> 
> No, the trouble is that POPL and conferences like it simply rejects
> too many fine, publishable papers.  This is bad because
> 
>  - Authors are badly served, obviously
> 
>  - Readers are badly served, because they don't get to read 
>    those papers
> 
>  - The papers get recycled at other conferences and workshops, where
>    they increase reviewing load (by being reviewed a second time),
>    and crowd out the truly workshop-y work in progress that should be
>    showing up at workshops
> 
> In short, we should just accept more papers at all our premier
> conferences, using a *quality* bar (is this paper good enough?) not a
> *quantity* bar (is it one of the best 30?).  How can we do this?  The
> "fat proceedings" problem is getting less and less important as we
> increasingly use digital media.  Really the only difficulty is how to
> accommodate the presentations at the physical meeting itself. But this
> is a problem that could be dealt with in many ways.
> 
> One straightforward one is to have parallel tracks.  Another is to
> have more days.  Still another, which I am rather fond of, is to
> accept (say) 60 papers, and then hold a lottery for 20 presentation
> slots.  [That's fewer than usual, so there'd be longer breaks for
> mingling, which is actually the real reason most people go to
> conferences in the first place.]  I would argue *against* choosing the
> "best" papers for presentation, because that will just re-introduce
> the ills we are currently struggling with.  Make it clearly a matter
> of luck, then no one will read anything into the "chosen for
> presentation" badge.
> 
> The lottery selection could even done at the conference itself.  I'm
> only half joking; that way, no one could be denied travel funding on
> the grounds that his or her paper had not been chosen for
> presentation.  Or perhaps participants registering for the conference
> could vote in advance for which accepted papers they'd like to see
> presented, so the programme is partly created by those attending the
> conference?
> 
> 
> A big advantage of this approach (simply accepting more papers) is
> that it is something we can simply choose to do.  It does not require
> every conference to make the same choice simultaneously, and it
> doesn't require a magical cultural change.  However, if we did take
> this path, then a significant cultural change would follow, over time.
> If publication at POPL was no longer an extraordinary achievement, but
> rather a recognition for a fine piece of work, appointment committees
> would in due course adjust their evaluation criteria.  And that in
> turn might actually reduce the overwhelming number of submissions to
> top-drawer conferences.
> Simon Peyton Jones
> 
> [1] J Wing, "CS woes: deadline-driven research, academic
>     inequality", CACM 52(12) Dec 2009, p8 
> 
> [2] J Crowcroft, S Keshav and N McKeown, "Scaling the academic
>    publication process to internet scale", CACM 52(1), Jan 2009,
>    pp27-30.
> 
> [3] M Vardi, "Conferences vs journals", CACM 52(5), May 2009, p5
> 
> [4] K Bierman, FB Schneider "Program committee overload in systems",
>    CACM 52(5), May 2009