[TYPES/announce] Two phase reviewing for POPL; a response

[Harry Mairson]

1.  Regarding the proposal of Simon Peyton-Jones to choose papers randomly from a list of papers judged worthy by a program committee:

Last year I was talking to a college president in the Boston area (not mine) who told me exactly the same thing about prospective freshmen---the university should just select those who are above threshold, and then choose randomly.  College admission is an increasingly crazy business in the US.  And for sheer size, it's a bigger problem than POPL.

Similarly, a close colleague of mine at another institution said to me about faculty searches: once the list gets down to 10 or 15 for a single position, it's a game of chance, even though there nominally continues to be a rational process.  What Simon is suggesting is that we just put up front the meritocratically random endgame that we may already have.

2.  Regarding the proposed increase in number accepted papers:

I fully understand and appreciate the rationale, especially as it relates to tenure and promotion.  The acceptance of a paper at POPL (among other conferences) is a valued achievement in those decisions, significant to institutions and crucial to individuals.  In what way is the needed achievement devalued when the number of accepted papers is increased?  Will promotion committees say, "That's a POPL paper from 201x, when they increased the size of the conference, and is not as significant as papers from 200x"?   That likely analysis suggests that the number of accepted papers be increased slowly---doubling is probably not a good idea.  

Recognize that people on committees count because they do not understand.  I ran a promotion committee (for someone outside CS) where I began the meeting by saying to its members (also outside the field), "What did this person do?  Maybe we should try talking about that first."  The discussion was a bit awkward---we all sounded like freshmen.

As long as there is a nominally meritocratic process for tenure and promotion, it will involve scaling some sort of pyramid.  The scaling and selectivity is what has defined the success: these conferences, grants, editorships, etc.  It may be that we have created, instead, an obelisk.  Are senior researchers who publish serially in these conferences conferring the value, or unduly increasing the pyramid's slope?  What's needed is moderation.

3.  Regarding the two-tier review process and the quality of reviews:

I agree with Matthias Felleisen---I've not seen much of Simon's "gold dust".  The quality of mercy is not strain'd, but the quality of reviewing is not improv'd.   We've all received (and probably written...) reviews where we immediately recognize, "they didn't get it", and we all know the boilerplate that comprises these reviews.  (I've found myself wishing that these reviews simply said, instead, "I didn't understand it and I didn't like it and I'm not interested.")  

The best critical reviews I've received are from colleagues who were trying to solve the same problems I was working on.  Getting boxed by people who know their stuff is an educational experience in itself.  When the solution of such challenging problems is a prize, then the local experts weigh in, sometimes even grudgingly, with their careful assessments---almost like sociologists of science---with their precise knowledge of what qualifies as a solution, what's winning, what's losing, what's cheating, and what's the canonical toy problem that needs solving to demonstrate that the solution is the right one.  They really know (and sometimes covet) those square centimeters of the intellectual territory---most reviewers don't.  The best compliment I've given colleagues is, "I wish I'd done that---and I tried."

Harry Mairson

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.seas.upenn.edu/pipermail/types-announce/attachments/20100112/96ba9868/attachment.htm