[TYPES] Two-tier reviewing process

[Dan Ghica]

On 29 Jan 2010, at 14:28, Matthias Felleisen wrote:

> Of course, Benjamin's and Steve's discussion about Expertise vs
> Confidence
> indicates that we simply have way too little time to evaluate  
> conference
> submissions fairly and robustly -- so why does this community insist  
> on
> using this brittle and unreliable mechanism to evaluate scientific  
> work?


I think the lack of time is only one problem.

For me there are two problems at least as serious:

1. Omitting proofs because of 'lack of space'. I see no reason why the  
PC does not demand that all details of all proofs are included in an  
appendix, not to appear in the printed proceedings but to be part of  
the refereeing process (perhaps even to be included in the online  
proceedings). Maybe we cannot check the proofs in details, but we can  
check a lot of things about the proofs, especially their existence.  
There are important papers out there, with high citation counts and  
fairly important results where the full proofs have never appeared.  
This is a problem.

2. Non-reproducible experiments. I had to referee papers about tools  
that could not be downloaded running on examples that could not be  
released. Even if the experimental data looks interesting, is it  
meaningful? This can be solved as well. For a paper that I submitted  
once I set up a virtual machine image with my tool running, loaded  
with all the examples, which could be reproduced at the push of a  
button. It's not that hard.

Thanks to all the contributors for an interesting a rather therapeutic  
discussion.

Dan


Dan Ghica
Lecturer / ARF
d.r.ghica at cs.bham.ac.uk