[TYPES/announce] Reviewing for POPL: a concrete proposal

[Simon Peyton-Jones]

Gentle colleagues

There has been a vigorous debate on the Types mailing list about
acceptance rates and criteria for POPL.  Phil Wadler, as SIGPLAN
Chair, and a member of the POPL steering committee, asked me to
present a concrete proposal for discussion at the POPL community
meeting tomorrow.  

Thank you for the opportunity, Phil.  The proposal appears below.  
I am sorry that I am missing the meeting this year.  Enjoy POPL!

Simon

Reviewing for POPL: a concrete proposal
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I will not repeat the details of the debate here, since you all
have access to it:
http://lists.seas.upenn.edu/pipermail/types-announce/2010/thread.html#1740

Instead, responding to Phil's request, I want to make single proposal

  That we use a "quality bar" rather than a "quantity bar"
  to govern acceptance for POPL.

It is difficult to quantify just what "POPL-publishable quality" is,
but I propose that it should be a level that, if it had been applied
in recent years, would have resulted in an acceptance range in the
region of 30%.  The current norm is 16-23%.  One data point is that
ICFP typically accepts rather more than 30%, but the quality IMHO is
still very high. (Historical figures for POPL, PLDI, and ICFP appear
below)

An alternative would to continue with a quantity bar, but increase it
substantially, say from its current 35 to 50 papers.  Personally I
prefer a target acceptance rate because my gut feel is that the
average quality does not change much year to year, whereas submission
volume does.

Regardless of the exact figure, I am advocating a sea change in our
attitude to the POPL evaluation process, not just an incremental shift
in policy.  

It is worth noting anecdotal evidence that individual POPL program
chairs have tried and failed to get their committee to accept more
papers.  The idea is discussed, potential papers are brought up, but
they are ultimately rejected.  To achieve this change, if we want it,
will take a broad community decision that gives a clear mandate to the
program committee.

With all that said, we can't tie the program committee's hands
completely, by requiring them to accept N papers or X% of submissions;
in the end we have to trust the PC.  This proposal is not to restrict
their discretion, but to give them a mandate.

Reasons for this change (in brief)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Fine, publishable papers are being rejected, which is bad for both
  authors and audience.  Those papers are recycled at other
  conferences and workshop, where they increase the reviewing load (by
  being re-reviewed) and crowd out workshop-y papers.

* Acceptance or rejection has a significant element of chance: it is
  very difficult for program committees to choose 35 out of 70 very
  good papers.  Yet, partly because it is so competitive, acceptance
  at POPL has a strong effect on promotion and tenure committees.
  Having career-important decisions based on a chancy process seems
  wrong.

* The pressure for slots makes it hard for a program committee to
  accept a ground-breaking but flawed paper over a more incremental
  but well-executed one.  This is not a clear-cut issue, but many
  people (including me) think that the relentless pressure for
  slots forces program committees to err towards more conservative
  conference programmes.

* It is a change that we can deliver.  In contrast, arguing that
  journal publications should be valued more highly might be
  desirable, but is a cultural change that no one can guarantee to
  deliver.  (However if POPL starts accepting more papers, a cultural
  change may well follow in due course.)

Consequences of the change
~~~~~~~~~~~~~~~~~~~~~~~~~~
If this proposal were to be accepted, we would need to figure out how
to accommodate many more papers at the physical meeting.  How to
achieve this is secondary to my main proposal, but a number of
proposals have been floated, including

* Parallel sessions
* A lottery among accepted papers
* Voting by conference registrants
* Program committee decision

I suggest that we invite the POPL steering committee to consider
these and other possibilities, and make a proposal in due course.


Background acceptance rates for POPL, PLDI, ICFP
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Source: http://people.engr.ncsu.edu/txie/seconferences.htm
	
---------------------------------------------------
Year      POPL           PLDI            ICFP
---------------------------------------------------
2008 	35/212(16%) 	34/184(18%) 	29/87(33%)
2007 	 36/198(18%) 	45/178(25%) 	26/103(25%)
2006 	33/167(20%)  	36/169(21%) 	24/74(32%) 
2005 	31/172(18%)  	28/135(21%) 	26/87(30%) 
2004 	29/176(16%)   ?/?(20%) 	21/80(26%) 
2003 	24/126(19%) 	28/131(21%) 	24/95(25%) 
2002 	28/128(22%) 	28/169(17%) 	24/76(32%) 
2001 	24/126(19%)  	30/144(21%) 	23/66(35%) 
2000 	30/151(20%) 	30/173(17%) 	24/110(22%)
1999 	24/136(18%)   26/130(20%)  	25/81(31%) 
1998 	31/175(18%)  	31/136(23%) 	30/70(39%) 
1997 	36/225(16%) 	31/158(20%) 	25/78(32%) 
1996 	34/148(23%)  	28/112(25%) 	25/83(30%) 
1995 	?  	       28/105(27%) 	? 
---------------------------------------------------