[TYPES/announce] POPL Review Process

[Martin Rinard]

  I am in complete agreement with Simon's proposal to increase the
number of papers accepted at POPL (and other programming language
conferences). I would like to add some reasons why I believe the field
as a whole would be better served by accepting more papers.

1) The current very selective process makes it difficult for program
   committees to accept papers that go against the existing value
   system of the field.  Such papers will almost always have at least
   one strong detractor, which makes the paper virtually impossible to
   accept over papers that are well-executed but more conservative
   (and therefore present less provocative ideas).

   The immediate negative impact of this situation is that the field
   loses the benefit of the new ideas in the rejected paper.

   A more subtle but arguably more pernicious effect is that
   researchers are motivated to perform relatively incremental
   research that reinforces the existing value system rather than more
   adventurous research that challenges the value system. I would not
   be surprised at all if this motivation is currently skewing the
   research that people in the field choose to do. An alternate, and
   potentially much more damaging possibility, is that people who are
   interested in innovation, creativity, and challenging the status
   quo choose not to enter the field in the first place.

   The overall effect is to suppress innovation and hamper the
   intellectual development of the field over time.  This is bad for
   the field both internally and as it competes with other fields.
   Based on my interaction with colleagues in a variety of situations,
   POPL is viewed as publishing technically excellent but conservative
   papers. This perception (which I believe has some justification in
   reality) is bad for the field - I have seen it negatively impact
   the perception of researchers who publish in POPL. 

2) My first four submitted papers (two of which were POPL papers) were
   all accepted. Anecdotally I have observed a similar pattern with
   many other students who chose to pursue research careers.  I
   believe that this early positive experience with the field helps
   students view the whole research endeavor positively and motivates
   them to stay in the field.

   The current publication process winds up rejecting many papers that
   are at least as deserving of publication as the papers that are
   accepted. Students who experience this kind of unjustified
   rejection can easily come to view the system (again with some basis
   in reality) as unfairly and arbitrarily failing to recognize
   legitimate accomplishment. As a result they become disillusioned
   with research and choose to pursue careers in other areas.
   
   By the way, I am not alone in this perception - I have heard other
   senior faculty express their frustration with the effect that the
   current publication process has on their students.

3) Greg Morrisett makes the point that rejected POPL papers are often
   published in other conferences. This is true, which mitigates some
   of the negative effects, but it is hardly an ideal
   situation. First, the publication delay denies the field timely
   access to the idea. Second, the publication delay runs the risk of
   getting the priority for the idea in the paper wrong. Finally, the
   proliferation of conferences fragments the field and hampers the
   useful interchange of ideas between subcommunities.

4) Rajeev Alur (and Simon Peyton-Jones, although he comes to a
   different conclusion) makes the point that selectivity can enhance
   both the reputation and visibility of the accepted papers. I agree
   with this point and indeed, believe that I have benefited
   enormously from this phenomenon myself. The problem is that the
   field has tipped over into a situation in which there are not
   enough slots in the top conferences to publish clearly deserving
   papers. There are two negative effects:
 
   a) The process becomes arbitrary and unfair because it becomes next
      to impossible to separate the top, say, 40 papers from the next
      20 papers based on merit.

   b) The selection criteria become biased to counterproductively
      emphasize execution and conformance to the existing value system
      over innovation and creativity, with long-term negative
      consequences for the development of the field as detailed above.

   In my view these negative effects, at this point in time, more than
   counterbalance the positive effects of an extremely selective
   system. Another way to look at this is that the growth in the field
   (over the last two decades) has outstripped the growth in the
   number of papers published at POPL, to the point that POPL is no
   longer an effective publication outlet for the field as a whole.

  So why is accepting more papers the right way to attack this problem
at this time? Because it effectively addresses so many of the problems
above. First, it removes some of the selection pressure and makes room
for more innovative papers in the short run and more innovative
research programs in the long run. Second, it provides a fairer
publication process that recognizes accomplishment and makes the field
more attractive to students who are looking for such a field. Third,
it maximizes publication timeliness. Finally, it would enable (but
certainly not require if this were not to be perceived to be
desirable) the creation of fewer but broader conferences that help
facilitate communication across subfields.

  In the long term, of course, the most productive course is to move
to a more standard conference/journal system in which the purpose of
conferences is to enable the fast dissemenation of recent information
and promote interaction between members of the community, while the
purpose of journals is to publish vetted results. But this is a topic
for a different discussion.

  Martin