[TYPES/announce] Two phase reviewing for POPL; a response

[Alessio Guglielmi]

Hello,

At 17:20 -0500 16/1/10, Michael Hicks wrote:
>>It happened to me to review a short and very well written paper 
>>(for a journal) whose claims looked to me absolutely impossible, 
>>outrageous. [...] I wonder what I would have done of that paper in 
>>40 minutes.
>
>I don't know the paper, but I disagree with the implication that 
>great papers will somehow be precluded from publication indefinitely 
>if they are not immediately accepted. [...] Indeed, revision may be 
>required to make the paper great.

Actually, I didn't mean such an implication, and, as I said, the 
paper was brilliantly written. As a reviewer, I didn't ask for 
changes in the exposition, because it was already perfect.

The paper was in proof theory/proof complexity, so, not really in the 
POPL scope, but still in computer science, and in particular in the 
range of interests of the TYPES list. The main reason why I'm 
discussing this issue here is because TYPES and my area intersect, 
and am worried by what I perceive as a cultural bias.

>As a rule of thumb, I've found that if I can't understand the basic 
>idea and why it works, the novelty of the idea, and its potential 
>impact within 40 minutes, there's reason to believe that POPL 
>readers would be similarly vexed, give up on the paper, and thus get 
>little from it.

I agree with what you say, except for one thing, the `why it works', 
and this is a very important point. So, let's use the example of our 
paper.

One could understand in one minute what the paper was claiming, its 
novelty and its impact, and 40 minutes would suffice to understand 
the basic idea, in broad terms.

However, its results would clash dramatically with my expectations, 
and indeed those of my community. This is one of those cases where 
one says `it cannot be true'. The only way to judge is to check the 
details, and this takes days or weeks of study because there's some 
serious mathematics behind.

Suppose this masterfully-written paper is submitted to a 
40-minute-review conference: should we accept it or not? I'd say YES 
if the conference is just for making ideas circulate and be 
discussed; I'd say NO if a conference accept is an important medal to 
be exhibited before a tenure committee, because we should make sure 
that tenured faculty actually prove what they claim.

(Probably, and rightfully, in our paper's case, the author didn't 
even dream of sending his paper to a conference.)

I suppose you see my point: 40-minute-reviewing is OK to judge 
whether an idea should be communicated, it's not OK to judge whether 
somebody is a good scientist (at least when some deep theory is 
involved). Unless, of course, we think that publishing at conferences 
is sufficient to define the good scientist. I think most disagree 
with it, but are we all aware that this is precisely the risk we are 
incurring? Isn't it obvious that our species is already perversely 
adapting to the conference environment? That the rule makers are more 
and more the products of the game?

If the computer-science peculiar way to conferences gains even more 
ground, the important kind of research of our paper will be even more 
discouraged than it is already (which is a lot). We will also see 
more conference papers brilliantly communicating false results, and 
people making careers out of that (perhaps not at the top places, but 
below?).

Expanding POPL (for example) means expanding the overall space for 
papers, because the expansion will trickle down the conference chain. 
This, unless the last conference down the chain closes (I bet it 
won't). Since nature abhors vacuum, more papers will be written; but 
we all are already at the limit of our working possibilities, so 
there will be even less serious reading.

(By the way, I agree with Peyton-Jones that writing is tremendously 
important for clarifying ideas, but this doesn't mean that after 
writing a paper the best thing to do is to rush it out. The best is 
to rewrite, and then rewrite, certainly not until perfection, but 
definitely until maturity.)

Anyway, expanding POPL means lowering the standards, plain and 
simply, perhaps not of POPL, but of the whole business. And what 
would the reason be? Because some are randomly denied the top medal? 
If conferences were just for communicating ideas, instead of being 
races, this would not be felt as much of a problem.

This is not a problem of POPL, of course, but of the whole culture at 
play here. Let's say that some of its assumptions are politically 
incorrect towards the minority for which leks, as Prakash accurately 
called them, are not suited.

Ciao,

-Alessio