[TYPES] Two-tier reviewing process

[Matthias Felleisen]

Dan, not to be contrarian, but how would the submission of 
additional materials solve the lack of time problem that 
prevents high-quality reviewing in the first place? (Just
to clarify: I actually accept that some submissions get high
quality reviews at some conferences. My problem is with the
'average' of reviews not those few who deliver outstanding
work under this extreme circumstances.) 

-- Matthias






On Jan 29, 2010, at 9:55 AM, Dan Ghica wrote:

> 
> On 29 Jan 2010, at 14:28, Matthias Felleisen wrote:
> 
>> Of course, Benjamin's and Steve's discussion about Expertise vs
>> Confidence
>> indicates that we simply have way too little time to evaluate conference
>> submissions fairly and robustly -- so why does this community insist on
>> using this brittle and unreliable mechanism to evaluate scientific work?
> 
> 
> I think the lack of time is only one problem.
> 
> For me there are two problems at least as serious:
> 
> 1. Omitting proofs because of 'lack of space'. I see no reason why the PC does not demand that all details of all proofs are included in an appendix, not to appear in the printed proceedings but to be part of the refereeing process (perhaps even to be included in the online proceedings). Maybe we cannot check the proofs in details, but we can check a lot of things about the proofs, especially their existence. There are important papers out there, with high citation counts and fairly important results where the full proofs have never appeared. This is a problem.
> 
> 2. Non-reproducible experiments. I had to referee papers about tools that could not be downloaded running on examples that could not be released. Even if the experimental data looks interesting, is it meaningful? This can be solved as well. For a paper that I submitted once I set up a virtual machine image with my tool running, loaded with all the examples, which could be reproduced at the push of a button. It's not that hard.