[TYPES/announce] Two phase reviewing for POPL; a response

[Philip Wadler]

Before the Types discussion began, I collected comments on the
two-phase proposal, which I planned to post on the web (with
permission).  I am now posting them here.   Enjoy!  -- P


------------------------------------------------------------------------

Dan Grossman, 11 Dec 2009

At first blush, I'm quite supportive of this.  Among the community I'm
one of the "happier" ones with how things are now, and quite concerned
about making major changes that "throw the baby out with the
bathwater".  But this proposal seems quite measured and certainly
worth trying as an experiment.  I agree it should lead to better
results.  It seems like a thoughtful compromise.

I had two questions that if I don't ask somebody else will, so I
encourage the POPL EC to think through rough answers:

1. Approximately what percentage do they envision being accept,
resubmit, and reject?  5% resubmit and 50% resubmit are _very_
different things.

2. Will there be a length limit on the cover letters?  500 words?  3
pages? No limit?

------------------------------------------------------------------------

Erez Petrank, 13 Dec 2009

As the proposal mentions, this is a step towards making our
conferences something of a journal, and probably another step towards
reducing the value of journals in our community. The discussion on the
POPL proposal may fit well into our planned discussion on conferences
versus journals in the next [SIGPLAN EC] meeting.

--Erez

------------------------------------------------------------------------

Manuel Chakravarty, 14 Dec 2009

Hi Phil,

This is an interesting proposal, and I'd be curious to see how it
performs in practice.  I wonder whether you have thought about any
specific means to evaluate both the amount of additional work
generated by two-phase reviewing (once it has been trialled) and to
evaluate its impact on the POPL program.  I appreciate that especially
the latter is hard to measure, which is why I'm curious whether any
concrete metrics have been discussed.

The proposal mentions having two PC chairs to spread the work load.
Do you also plan to have more PC members on the committee?  How
detailed do you expect the second-phase reviews to be?

Cheers,
Manuel

------------------------------------------------------------------------

Francois Pottier, 14 Dec 2009

Dear Phil,

Some thoughts and questions:

 + the idea that this would work better than the current author
 response process sounds plausible. Is there an evaluation of the
 current rebuttal process?

 o one should make sure that the two weeks during which authors are
 supposed to revise the paper do not fall within the summer break
 (i.e. avoid August; early September would be fine)

 o how are the reviewers supposed to evaluate the revised version?
 read it all again in detail? have a quick look? will at least one new
 reviewer be enrolled?

 - if the second round reviewers are the same as the first round
 reviewers, then it is not clear that the quality of the selection
 process will improve.

 - the review process will become significantly longer (at least for
 the papers which pass the first round). It is not very encouraging
 for authors to have to spend several months waiting for the final
 result. It is perhaps more acceptable for a journal, where presumably
 the work is well polished already and stands a good chance of being
 accepted; but for a selective conference, where there is a good
 chance of being rejected, isn't that a waste of time? If conferences
 are supposed to encourage the dissemination of new results, then the
 process should be quick.

  (One might argue that the authors should not wait, but instead
  continue working while the paper is under consideration. Maybe so,
  but this is a shift to a new paradigm.)

--
François Pottier
Francois.Pottier at inria.fr
http://gallium.inria.fr/~fpottier/

------------------------------------------------------------------------

James Hook, 14 Dec 2009

I think it is a very good idea to have conferences experiment.  I
would like to thank the POPL steering committee for creating this
proposal, and for inviting feedback.

On the goal: I think the goal "improve the decision process" needs
refinement.  What aspect of the decision process do you wish to
improve?  Is there any way to measure it?  Do you feel the current
process is too conservative, and want to encourage publication of less
incremental work?  Do you feel that too many technically weak papers
have been accepted?  Do you feel that too many technically sound and
subsequently influential papers have been rejected?  Do you feel that
authors have not taken the opportunity to improve their papers once
they are notified of acceptance?  Are authors of rejected material
getting reviews of sufficient quality to help them improve their work?
(There is some discussion of these points in the "advantages section",
but I would recommend a clear goal statement.)

Leading the proposal with the impact of the decisions on promotion and
tenure cases suggests to me that perfection might be a goal.  The
statement is true---the decisions have consequences.  But ultimately
the PC must evaluate papers, not individuals, and make the best
decisions they can on the work products submitted.  We serve our
community best by bringing integrity to the reviewing process, not by
allowing ourselves to be distracted by the very real personal
consequences of our decisions.

I think perfection of the process should not be a goal.  Experience
suggests that there will be some clear winners, some clear losers, and
a bunch of papers in the middle.  I have seen no evidence that any
amount of time invested in reviewing or discussing the papers on the
cusp will significantly improve the quality of the technical program
selected.

Overall structure: I would recommend structuring the proposal as an
experiment.  What is the goal?  What is the evaluation plan?  When
will it be evaluated?  I doubt we would learn very much from doing it
once and then abandoning it.  After it has been tried and tuned for
three years is it ready to be evaluated?

Timeline: The proposal is trying to balance end-to-end time with time
for meaningful revisions.  I think this is an area where we need to
gain some experience to make an informed judgement.  The
submission-to- presentation time seems quite long to me.

It is difficult to predict what the impact will be of having the POPL
submission date coupled to the ICFP notification date.  Currently the
ICFP satellite events are informally coupled with the ICFP
notification date.  The largest of these is the Haskell Symposium.
Perhaps the best thing to do here is to include some measurement of
ICFP workshop and Haskell symposium submissions as part of the
evaluation plan.  It is clear that it is not the intent of the
proposal to weaken these technical meetings, but it is one possible
consequence.  I hope other members of the ICFP SC will comment on this
aspect.

Other issues: To what extent is the current print publication process
adding confounding delays to the overall submission-to-presentation
time?  Is there a way to borrow time from Sheridan instead of moving
the submission deadlines earlier in the year?

In closing, thank you for the stimulating and innovative proposal!

Regards,
Jim

------------------------------------------------------------------------

Martin Abadi, 6 Jan 2010

Dear Phil,

I am concerned about the possibility of revisions, for two reasons:

* It can be a scheduling problem. Not all authors will be equally available
in any two-week period, and some periods are particularly harsh on
particular groups of authors (e.g., the French in august, academics near the
start of the quarter).

* I am unclear on what sorts of revisions will be expected or allowed, and I
am concerned that this will be a source of problems when working on a tight
schedule.

Happy New Year.

Martin

------------------------------------------------------------------------

Cristina Cifuentes, 7 Jan 2010

It is not clear to me what the net result of the two-phase reviewing
will be: to discourage journal publications even further?  Or to
increase the quality of POPL papers?

POPL papers have a page limit, which journals do not normally have.  A
journal paper is more complete, and as such, I believe journals are a
good place to publish more complete work that cannot "fit" into a
conference paper.

The two-phase review will improve the quality of some POPL papers at
the expense of more work by the PC.  POPL already has a high standing
in academia, increasing its quality will NOT make it the same as
publishing in a journal.

In other fields, journal papers are highly valued and are rather
short, e.g., Nature (4 page papers).  One could argue that part of the
problem with our journal papers is that they are too long: long time
lead to write, review and publish, leading to not-up-to-date
information.  In fields like Nature, papers are published in a timely
basis, but their length is rather short.

Regards,
Cristina

------------------------------------------------------------------------

Derek Dreyer to Benjamin Pierce, 8 Jan 2010

Based on this email, I don't entirely understand the proposed process,
nor do I really understand its motivation.  What is the problem being
solved?  It seems that the idea is to make POPL reviewing more similar
to journal reviewing.  However, it's unclear to me what "resubmit"
corresponds to in journal reviewing.  "Resubmit" on a journal
submission usually is tantamount to "major revisions needed", but
presumably major revisions cannot be carried out in 2 weeks.  Or is
that not the meaning of "resubmit"?

After the PC initially decides between accept, resubmit, and reject,
who has to resubmit?  Is it just the people who got "resubmit" who are
judged in the second round?

Why does the PC meet physically after the first round but not the
second?  Is it because it is expected that there will only be a few
papers in the "resubmit" category, that it will be used sparingly for
only really tough borderline cases?  Otherwise, I can imagine a PC
meeting devolving into "resubmit everything".

One very real concern I have is that the people who would be most
likely to resubmit their POPL rejects to ESOP (which would perhaps be
among the better papers at ESOP) are precisely those who would have to
wait an extra 6 weeks to get "reject" and thus have missed the ESOP
deadline.  I guess it's not POPL's concern whether it feeds into ESOP,
but I thought I'd mention it.

------------------------------------------------------------------------

Peter Sewell to Benjamin Pierce, 8 January 2010

I'm strongly against this proposal. The current process certainly has
a lot of noise, but I don't think this would be much better - much of
the noise comes from the PC and reviewer assignment at the start,
despite the best efforts of the PC chairs.  This proposal sounds like
a massive amount of additional work for the PC and (if there are a
significant number of "resubmits") for the reviewers and authors.
Instead, as there often seem to be many perfectly good papers that get
rejected, I would simply move to accepting more papers, multi-tracking
or extending the conference.  It might also be that more care to
ensure a subject-balanced PC could be taken.

------------------------------------------------------------------------

John Reppy to Kathleen Fisher, 10 January 2010

On the face of it, I think that this is a bad idea that will increase
the work for reviewers, who are already overworked.  It is not clear
what problem this proposal is attempting to fix.  Is there a concern
that bad papers are getting accepted to POPL by accident?  I suspect
that that is not a serious problem.  Is the concern that the quality
of POPL reviewing is not high enough?  That could be addressed by
increasing the size of the PC (or perhaps using the external reviewer
mechanism of PLDI) to reduce the number of reviews that each member is
responsible for, but again I suspect that review quality is not a
pressing issue (in fact, the quality of reviews has significantly
improved over the time that I've been submitting conference papers).

Instead, the problem that POPL suffers from (and it is not alone in
this regard) is an insufficient number of slots for the number of
worthwhile papers that are submitted.  The proposal does not address
that issue at all.  The fact that POPL is both a crap shoot and really
important for tenure decisions is a symptom of a broader problem that
our discipline faces.  Our current publication model doesn't really
scale to an ever-growing community of researchers.  In the short term,
we can increase the number of slots by decreasing talk length
(probably a good thing) and adding parallel sessions (probably a bad
thing).  I do worry that increasing the conference size will
eventually reach a point where there won't be consistency across the
reviewing process.

I am also concerned about the first-round accept class of papers.
What percentage of papers would likely be accepted/rejected the first
round?  Can a paper that is accepted the first round be rejected in
the second round?  Is this just a mechanism to force authors to make
the changes requested by reviewers?

This plan also loses what, I think, is the most important benefit of
author response.  Namely, that it is a forcing function that requires
reviewers to produce a quality review before the PC meeting.

Lastly, I do think the idea of moving to co-chairs for the PC has merit.

------------------------------------------------------------------------

Karl Crary, 11 Jan 2010

I won't be at POPL this year, so I thought I would send you my comments:

I'm not against the proposal, exactly.  I can see some possible
benefits, and it definitely seems like an improvement over the current
system of author response (which I am against).  However, I am
troubled by the endless tinkering with a system that seems to be
functioning well overall.  Each change carries with it the possibility
of some negative unintended consequence, while the upside is limited.

I had hoped that the system had already converged, and I wouldn't mind
rejecting the proposal on the if-it-ain't-broke-don't fix-it
principle. But if we do adopt the proposal, I hope it's the last
change for some time.

I do appreciate you raising the topic for general discussion.  It's a
welcome contrast to the recent history of unilateral changes.

 -- Karl

------------------------------------------------------------------------

-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.