[TYPES/announce] Request for comments: Two-phase reviewing for POPL

[swarat@cse.psu.edu]

Request for comments: Two-phase reviewing for POPL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Though program chairs and committee members work hard to get things
right, no decision process is perfect.  Given the role that major
conferences such as POPL play in promotion and tenure, the system is
coming under increasing pressure and discussion.  (For instance, see
'Conferences vs journals in computing research', Moshe Vardi, CACM,
May 2009.)

The POPL Steering Committee has formulated the following proposal,
which we are circulating for discussion and feedback from the
community.  The proposal aims to improve the decision process for POPL
while still working in a fixed time frame and with bounded resources.

We propose to use a two-phase reviewing process, broken up
approximately as follows:

(a)  8 weeks:  Reviewers submit first-phase reviews
(b)  2 weeks:  On-line discussion
(c)  2 days:   Physical PC meeting
 *** Decide for each paper: accept, resubmit, or reject ***
(d)  2 weeks:  Author revise papers (for each accept or resubmit)
 *** Authors resubmit paper with cover letter describing changes ***
(e)  4 weeks:  Reviewers submit second-phase reviews
(f)  1 week:   On-line discussion
 *** Decide for each paper: accept or reject ***

The times may require tuning.  It is proposed to omit author response,
as two-phase reviewing serves a similar purpose.  Resubmitted papers
should include a cover letter describing what has changed and responding
to concerns raised in the reviews.

We also propose that the program committee should have two co-chairs.
Two co-chairs provides additional effort to ensure that papers receive
expert reviews and to manage the extended review process.  As the team
for POPL 2011 is already in place, we expect to first try two-phase
reviewing for POPL 2012.

We envision that the new, earlier submission date for POPL should
come, if possible, about two weeks after the ICFP notification date.
We plan to survey author satisfaction, starting with POPL 2010, to
provide some feedback on the process.

Advantages of this proposal include:

* Revision is likely to improve the quality of the papers, and in
particular may improve readability of the final result.

* Revision provides a sounder footing than author response for
dealing with papers about which there is doubt.

* Some argue that POPL receives more high-quality papers than it can
accept.  Improving the review process may provide a better basis for
deciding whether to increase the number of accepted papers.

* Other conferences are moving to a year-round refereeing process
closer to that used by journals; for instance VLDB is now linked to
a journal PVLDB.  The two-phase proposal yields similar benefits,
while ensuring focus and a bound on effort.

We seek your comments!  Please speak or write to any member of the
POPL Steering Committee.  There will be a community meeting at POPL,
5:15-6:30pm Wednesday 20 January 2010, to discuss this plan.

  * Philip Wadler, current SIGPLAN Chair and 2008 Program Chair
  * Kathleen Fisher, past SIGPLAN Chair
  * Graham Hutton, current SIGPLAN Vice Chair
  * Chandra Krintz, past SIGPLAN Vice Chair
  * Tom Ball, 2011 General Chair
  * Mooly Sagiv, 2011 Program Chair
  * Manuel Hermenegildo, 2010 General Chair
  * Jens Palsberg, 2010 Program Chair
  * Zhong Shao, 2009 General Chair
  * Benjamin Pierce, 2009 Program Chair
  * George Necula, 2008 General Chair
  * Martin Hofman, 2007 General Chair
  * Matthias Felleisen, 2007 Program Chair