[TYPES/announce] Two phase reviewing for POPL; a response

[Michael Hicks]

> It happened to me to review a short and very well written paper (for
> a journal) whose claims looked to me absolutely impossible,
> outrageous. After reading (= studying) the paper for days, I still
> was in the dark, and had to dig out some older results the paper was
> relying upon, and this took me weeks of study. The paper was right,
> it was accepted, and it changed dramatically my own research ever
> since. A fantastic idea. I wonder what I would have done of that
> paper in 40 minutes.

I don't know the paper, but I disagree with the implication that great  
papers will somehow be precluded from publication indefinitely if they  
are not immediately accepted.  As Greg Morrisett previously noted,  
there are many other high-quality publication venues if the paper is  
not selected for POPL.  Indeed, revision may be required to make the  
paper great.

When I review a paper, I am interested in the technical idea in the  
paper *and* how well the paper communicates that idea to the reader.   
Indeed, these two things are tied together.  As a rule of thumb, I've  
found that if I can't understand the basic idea and why it works, the  
novelty of the idea, and its potential impact within 40 minutes,  
there's reason to believe that POPL readers would be similarly vexed,  
give up on the paper, and thus get little from it.  Understanding all  
of the technical details for the general case is another matter and  
will take longer; but the basic idea and approach should be clear  
enough after an hour.

I have been in conversations with people who have complained that a  
particular published paper is not very deep/interesting/insightful/ 
etc. but after some discussion realize there is more to the paper than  
they thought, but the paper fails to communicate it clearly.  Perhaps  
the paper could use better motivation, a few key examples, a careful  
description of an elided algorithm, better comparison to a related  
approach, ... whatever.  As a reviewer, I try to suggest where I'm  
having trouble understanding the paper, how it could make its points  
better, etc. so the authors can revise accordingly.

So while POPL PCs may not select eventually-great papers, I think it  
would be even more tragic for a paper to be published prematurely, and  
thus a potentially great idea not appreciated, especially when that  
idea could have been more widely understood after a round of revision.

-Mike