[TYPES/announce] Two phase reviewing for POPL; a response

[Matthias Felleisen]

Simon and others, 

thanks for the feedback. While I am a member of the POPL SC until 
January, this email is my personal opinion and is not to be 
construed as something anyone else on the SC may subscribe to. 


1. I like the idea of accepting more papers. It is a part of
every year's SC discussion. And as you probably recall from your
chairman role, the SC is quite encouraging about accepting 
more papers. I tried to get more papers into my POPL. This
year again the SC is encouraging the PC to broaden the 
acceptance range. 

In my own experience, though, the PC tends to block attempts 
to, say, double the number of papers. The idea is discussed, 
potential papers are brought up, and eventually the faction 
that disapproves of a larger program wears out those who are 
on the 'more papers' side. 

If you wish to achieve this numeric enlargement, I am afraid
you will need to set a target number, say 60 papers. 

2. I strongly disagree with your description of the state of
affairs:

> When I began my academic career an author would be lucky to get
>  three scrawled sentences of review, on physical scraps of paper.
>  Nowadays authors get between three and six substantial, thoughtful
>  reviews.  That is gold dust.

I started with the same kind of paper reviews. I do not see
much of the improvement you see. 

On some occasion, you get fantastic reviews (even for rejected 
papers). 

On many occasions, you get feedback from non-experts that is 
nearly random. I won't blame the committee members here: they 
are under time pressure, they have their own preferences, 
and they have the backgrounds they have. We are all weak, and 
we do what we can do. 

Over the past few years, I have collected informal statistics 
that basically suggests that submissions get about one expert
review per submission but that may mean two for one submission
and none for another. Yes, a good number of papers are rejected
without knowledgeable review. 

The expectation with a two-phase review is this: 

-- At the first, physical meeting papers w/o expert reviews 
are discovered. They get to respond to their preliminary reviews
and the PC will figure out how to get appropriate reviewers 
for these papers. No more 'over night, after dinner'
expert reviews. 

-- Papers with a fixable weakness are sent back to authors so 
that they can demonstrate clearly how to overcome this weakness. 
Hopefully this will help get some less-finished ideas
into the conference and turn the 'oral journal' that POPL currently
is into a real conference again. 

3. The two-tier proposal is a compromise that a dozen or so 
representative POPL community members have worked out. Like all 
compromises they don't make people 100% happy. I can only speak 
for myself; I am in the not-100% happy camp. 

But after discussing this issue for the entire fall, I believe that 
the (once again) suggested increase of acceptances and this 
proposal may get POPL moving in the right direction. I do consider it 
an experiment -- as I am sure others on the SC do, too -- that should
be evaluated in a few years. If it moves in the wrong direction, 
we go back to the status quo or we try a different change. 
It it turns out to be an improvement, we can still try other
modifications. 

-- Matthias