[TYPES] Two-tier reviewing process

[Norman Ramsey]

 > Following up on Benjamin's comments about "X" reviews.  There are two 
 > different axes that are important when understanding a review.  One is 
 > the reviewer's  *expertise* in the subject of the paper.  Another is the 
 > reviewer's *confidence* in his or her assessment carried out in the 
 > review.  Using only one score to indicate both leads to some confusion, 
 > since the two properties get conflated.  

I second this argument loudly.  It is a major flaw in the "Identify
the Champion" model that Oscar Nierstrasz has so widely publicized.
While "Identify the Champion" is far superior to what it replaced,
reviewers have a strong tendency to conflate expertise with
confidence.  The program chair can provide detailed guidelines to the
effect that A and D indicate high-confidence reviews, and that
low-confidence reviews should received B and C scores, but busy PC
members cannot keep these subtleties in mind.

The problem with "Identify the Champion" is that there aren't enough
independent axes for a review to express:

  - this is how much I know about the subject area
  - this is the degree to which I want the submission in (or out)
  - this is how confident I am in my recommendations

Those poor people who reviewed papers for me may remember receiving
rather complex, confusing instructions about the scores I was looking
for.  Here is a brief excerpt for the 'B' score, sometimes called
'weak accept':

    B: I think this paper should be accepted, but I am not sufficiently
       confident in my own judgment to want to argue for it at the
       meeting.  I am hoping someone more confident or more knowledgeable
       will make the case.

                      - or -

    B: I see both good and bad in this paper, and I don't want to take a
       strong stand.  At the meeting, if others want to accept it, great;
       if not, a good argument against it will either convince me to reject
       it or will help me discover that I want to argue for it after all.

I would be delighted if someone were to step forward and propose an
alternative to or refinement of ABCD XYZ.  God knows we have been
using long enough that collectively, we should be well aware of its flaws.


Norman

P.S. I agree with Benjamin not to take the fraction of 'X' reviews too
seriously.