[TYPES/announce] Proposed changes to POPL review process

[Derek Dreyer]

Hi, Phil.

FWIW, the POPL'11 PC also discussed this issue in detail when deciding
what kind of meeting we are going to hold this year.  As you say, the
issue split people down the middle.  I thought it might be useful to
summarize the main arguments that were given for physical vs.
electronic meetings.  (To clarify, by "electronic", I'm talking here
about a week-long, online discussion through the conference management
website, NOT a phone discussion, which most of the PC members
pooh-poohed.)


Main benefits of physical meeting:

- Easier to get a "global" view of the submissions

- PC members are more likely to contribute to the discussion of papers
they didn't review

- More effective use of PC members' "bandwidth", no delays in
communication due to time zone differences

- Excellent networking opportunity, esp. for more junior PC members

- Much more fun, esp. for extroverts


Main benefits of electronic meeting:

- No travel costs, no jet lag, no carbon footprint

- Longer, more careful and thorough discussion of papers (with the
possibility of requesting additional expert reviews during the
discussion)

- Easier, more fun for non-native English speakers (as well as
introverts) to participate


In the end, the POPL'11 PC decided to have a 10-day electronic meeting
followed by a 2-day physical meeting.

Personally, I started the discussion strongly in favor of physical
meetings, for all the reasons given above, but I found the arguments
for electronic meetings to be compelling, and I sense that the future
is electronic.

Phil, concerning your comment that the advantages of electronic
meetings become clearer in a context where more papers are accepted, I
understand the argument but I don't think it's so clear-cut.  You're
right that there's less of a need for a "global" view in this case,
because determining the "cut-off" point is less important.  OTOH,
getting a global view is still important for the purpose of
normalizing the reviewing process.

That is, one paper might get a B and 2 C's (and get rejected), and
another might get an A and two B's (and get accepted), with the only
difference being that the first paper had reviewers with low average
scores and the second paper had reviewers with high average scores.
To detect this problem, one cannot simply consider each paper
independently, and so I think it is easier to detect in a physical
meeting, given that more people pay attention to the discussion of
each paper.  But there may, of course, be purely electronic solutions.

Best regards,
Derek


On Sat, Jan 16, 2010 at 4:39 PM, Philip Wadler <wadler at inf.ed.ac.uk> wrote:
> [ The Types Forum (announcements only),
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>
> Following on from Rajeev Alur's e-mail, I want to make the following proposal:
>
>  - Replace the POPL physical PC meeting with an electronic one.
>
> This proposal was discussed by the POPL Steering Committee, which was
> split on the matter.  Input from the community would be helpful.
>
> My experience matches Rajeev's.  The process of a physical PC meeting
> necessarily requires quick decisions, often based on who is most vocal
> or an overnight reading of a paper.  My experience of electronic
> meetings is that the discussions can be more considered.  I believe
> the advantages of electronic meetings become clearer in a context
> where more papers are accepted, so the decisions centre on 'Is this
> paper of good quality?' rather than 'Is it better than the other
> papers we've accepted?'
>
> In addition to the cogent points raised by Rajeev, I'll mention that
> there is also the cost, particularly to the climate, of a physical
> meeting.  Typically, POPL has about 200 attendees and a PC of 25, so a
> physical PC meeting increases the carbon footprint of the conference
> by on the order of 10%.
>
> How does the community feel about a move to electronic PC meetings?
> Yours,  -- P
>
>
>
> On Thu, Jan 14, 2010 at 3:32 PM, Rajeev Alur <alur at seas.upenn.edu> wrote:
>> [ The Types Forum (announcements only),
>>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>>
>> Dear Phil,
>> It has been interesting to see all these responses to the two-phase
>> proposal. I also think two-phase reviewing is not that great.
>> One additional drawback I see is that getting rejection after the
>> second phase will amplify the frustration (particularly for starting
>> graduate students).
>> I feel that low acceptance ratio of POPL is a desirable feature, it is
>> critical
>> to its reputation, and I would hate to see that go up significantly.
>> As an aside, some people have essentially suggested that "since number
>> of POPL papers impacts tenure, we should make it easy for researchers to
>> publish in POPL". I do not find this argument compelling. In fact, I am
>> not aware
>> any committee counting this number. What matters is whether senior POPL
>> researchers are impressed by your work, and given that POPL is selective,
>> easiest way for someone to gain attention is by publishing
>> papers in POPL. Fortunately, selection is based on merit, so this presents
>> a clear recipe to draw attention to your work. If POPL is not selective,
>> then the only way would be to be a student of a famous advisor.
>> In fields such as control theory, top conferences such as CDC have
>> high acceptance rates, and indeed good pedigree is a necessity.
>> In any case, POPL review process should focus on selecting best papers
>> and maintaining high quality, without worrying about other factors.
>>
>> More constructively:
>> conferences such as LICS and CAV use electronic PC meeting.
>> I have been on POPL PC once and PLDI PC once, but I have a lot more
>> experience with LICS and CAV (also as PC Chair for both).
>> The problem with POPL (or LICS/CAV for that matter) initial reviews is
>> not the
>> quality (with some exceptions, most papers' contribution is very clear
>> from the reviews),
>> and also not the selective biases of individuals (which are a given, and
>> also, useful,
>> otherwise no evaluation would be possible), but rather that assignment
>> of letter grades
>> A/B/C/D in a distributed manner. For example, a clear technical advance
>> on a well-studied
>> problem may get a B or a C depending on the reviewer. This can make a huge
>> difference, and thus, the same paper rejected from one conference may
>> get accepted the
>> next time, making the process unpredictable. The goal of the PC meeting
>> is to correct
>> for this bias. But the physical meeting is not conducive to correcting
>> this.
>> For a given paper, the opinion of the person whose interests match most
>> closely with the paper,
>> counts more (but it should not: experts' reviews are useful,
>> not his/her biases on what to do with supposedly incremental, or supposedly
>> theoretical-that-will-never-work, or supposedly
>> practical-but-not-conceptually-deep papers).
>> Also, more vocal people get more influence. Time pressure impacts
>> decisions.
>> In practice, PC members are actively involved only in papers they have
>> been assigned,
>> maintaining the distributed nature of the process.
>> What one says on the spur of moment weighs more than what one writes
>> after careful
>> thought, editing, and sanity checks.
>> Thus, physical PC meeting adds unpredictable noise in the selection
>> process.
>> These are less of a problem in an electronic PC meeting. I think every PC
>> member needs to look at all the papers, and focus on selecting best X
>> submissions
>> based on reviews by applying his/her bias uniformly (and not just to
>> one's own pile).
>> This is easier to do on a longer time scale of electronic PC meeting.
>> Bottomline: not clear why POPL does not switch to electronic PC meeting.
>>
>> More dramatically:
>> I mentioned this to Jens after this year's POPL meeting: abolish the PC
>> (i.e. reduce its role
>> to a "reviewers committee" of an exapnded size).
>> Two or three co-chairs can collect reviews for each paper from those who
>> are real experts
>> on the subject. Then based on the reviews, make a decision applying fair
>> and uniform standards.
>> This is not as bad as it sounds. Jens indeed spent a lot of time
>> browsing through all submissions anyway,
>> and could have easily picked the papers after looking at the reviews.
>> Maybe a single person's bias would be detrimental,
>> but, say 3, would make the process better than it is now (and reduce the
>> cumulative amount
>> of time one would spend on POPL PC duties).
>>
>> best regards
>> --rajeev
>>
>>
>>
>>
>
>
>
> --
> .\ Philip Wadler, Professor of Theoretical Computer Science
> ./\ School of Informatics, University of Edinburgh
> /  \ http://homepages.inf.ed.ac.uk/wadler/
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>