[TYPES] Improving the quality of POPL reviews?

[Norman Ramsey]

Mike Hicks writes to ask that we not forget to consider the quality of
reviews we receive from POPL and from other SIGPLAN conferences.

Regarding the quality of the reviews I myself receive, I lie between
Simon Peyton Jones (gold dust) and Matthias Felleisen (not much
improvement).  I am often impressed by the number of reviews my
submissions receive, and I am usually happy enough with the quality.
Occasionally I get a real gem from a reviewer who teaches me a better
way to explain my work, but this is the exception, not the rule.

The stated goals of the (probably dead) proposal from the Steering
Committee did not include providing better reviews.  And while the
gradual review process described by Andrew Myers sounds attractive
(more and better reviews for better submissions), it's not obvious to
me how it might affect the quality of the reviews overall.

For those who might be interested in better reviews, I propose the
following cheap and cheerful experiment: during the author-response
period (which is now nearly standard), allow each author to rate each
review anonymously, on a 3-point scale:

  +1                 This review was clear and helpful
   0 (do nothing)    Typical review, not noteworthy
  -1                 I was disappointed in the quality of this review

Fearful authors could deposit their votes in escrow with a trusted
third party.  Aggregate data could be published in the proceedings.
Truly brave committees could make their scores public.

I expect that evaluation by authors would make reviews slightly
better, but that authors would quickly adjust their expectations, so
in the long run scores would be stable.  Reviews would be just
slightly better than they are now---not a bad outcome.

I hope that reviewers who were repeatedly and consistently downrated
would stop receiving invitations to serve on program commmittees.


Norman Ramsey