[TYPES/announce] Proposed changes to POPL review process

[Rajeev Alur]

Dear Phil,
It has been interesting to see all these responses to the two-phase
proposal. I also think two-phase reviewing is not that great.
One additional drawback I see is that getting rejection after the
second phase will amplify the frustration (particularly for starting 
graduate students).
I feel that low acceptance ratio of POPL is a desirable feature, it is 
critical
to its reputation, and I would hate to see that go up significantly.
As an aside, some people have essentially suggested that "since number
of POPL papers impacts tenure, we should make it easy for researchers to
publish in POPL". I do not find this argument compelling. In fact, I am 
not aware
any committee counting this number. What matters is whether senior POPL
researchers are impressed by your work, and given that POPL is selective,
easiest way for someone to gain attention is by publishing
papers in POPL. Fortunately, selection is based on merit, so this presents
a clear recipe to draw attention to your work. If POPL is not selective,
then the only way would be to be a student of a famous advisor.
In fields such as control theory, top conferences such as CDC have
high acceptance rates, and indeed good pedigree is a necessity.
In any case, POPL review process should focus on selecting best papers
and maintaining high quality, without worrying about other factors.

More constructively:
conferences such as LICS and CAV use electronic PC meeting.
I have been on POPL PC once and PLDI PC once, but I have a lot more
experience with LICS and CAV (also as PC Chair for both).
The problem with POPL (or LICS/CAV for that matter) initial reviews is 
not the
quality (with some exceptions, most papers' contribution is very clear 
from the reviews),
and also not the selective biases of individuals (which are a given, and 
also, useful,
otherwise no evaluation would be possible), but rather that assignment 
of letter grades
A/B/C/D in a distributed manner. For example, a clear technical advance 
on a well-studied
problem may get a B or a C depending on the reviewer. This can make a huge
difference, and thus, the same paper rejected from one conference may 
get accepted the
next time, making the process unpredictable. The goal of the PC meeting 
is to correct
for this bias. But the physical meeting is not conducive to correcting 
this.
For a given paper, the opinion of the person whose interests match most 
closely with the paper,
counts more (but it should not: experts' reviews are useful,
not his/her biases on what to do with supposedly incremental, or supposedly
theoretical-that-will-never-work, or supposedly 
practical-but-not-conceptually-deep papers).
Also, more vocal people get more influence. Time pressure impacts 
decisions.
In practice, PC members are actively involved only in papers they have 
been assigned,
maintaining the distributed nature of the process.
What one says on the spur of moment weighs more than what one writes 
after careful
thought, editing, and sanity checks.
Thus, physical PC meeting adds unpredictable noise in the selection 
process.
These are less of a problem in an electronic PC meeting. I think every PC
member needs to look at all the papers, and focus on selecting best X 
submissions
based on reviews by applying his/her bias uniformly (and not just to 
one's own pile).
This is easier to do on a longer time scale of electronic PC meeting.
Bottomline: not clear why POPL does not switch to electronic PC meeting.

More dramatically:
I mentioned this to Jens after this year's POPL meeting: abolish the PC 
(i.e. reduce its role
to a "reviewers committee" of an exapnded size).
Two or three co-chairs can collect reviews for each paper from those who 
are real experts
on the subject. Then based on the reviews, make a decision applying fair 
and uniform standards.
This is not as bad as it sounds. Jens indeed spent a lot of time 
browsing through all submissions anyway,
and could have easily picked the papers after looking at the reviews.
Maybe a single person's bias would be detrimental,
but, say 3, would make the process better than it is now (and reduce the 
cumulative amount
of time one would spend on POPL PC duties).

best regards
--rajeev