[TYPES/announce] Reviewing for POPL: a concrete proposal

[S. Doaitse Swierstra]

The POPL event is actually a multi-event, with a lot of workshops  
around one main conference with top papers. One of the problems with  
refereeing is that first papers are sent to POPL and once  rejected  
are resubmitted  as a workshop paper.

Notwithstanding the good work done by the POPL-PC's I know from my own  
experience that:
  - quite a number of papers submitted are probably not so excellent;  
but why not try since there always is a second chance
    at a more specialised event?
  - papers come form a large variety of subjects, and it is not always  
easy to find a sufficient number of informed PC members

I can see a solution in which:
   - people initially/only submit to one of the more specialised  
conferences and workshops
   - the PC's of these conferences select the papers they think are of  
high quality and deserve to be presented to a wider community because
     they represent something new and interesting for everyone
   - the POPL PC constructs a nice single track conference out of  
these preselected papers, and there are no conferences scheduled in  
parallel
     when these papers are presented (e.g. in the morning)
   - the other events run in the afternoon and in parallel with the  
papers which were accepted by their PC's except
     those who made to the plenary POPL sessions.

As a result we have more accepted papers, a better balanced program,  
automatic decisions about paralell session, and the committee has an  
easier job, since fewer and better papers have to be judged, and an  
initial review has already been done by the experts in the PC of the  
associated conferences. It also takes a bit of the gambling effect away.

 From my Haskell Symposium/ICFP experience I would rather have the HS  
PC select which are the best Haskell papers and send them one level up  
to the ICFP plenary forum, then to leave the selection to the ICFP PC.  
Given the broader scope of POPL I would expect this effect to be even  
stronger.

  Doaitse


On 20 jan 2010, at 11:35, Alain Girault wrote:

> [ The Types Forum (announcements only),
>     http://lists.seas.upenn.edu/mailman/listinfo/types-announce ]
>
>
> Dear all
>
> First, I completely agree with Simon's proposal.
> Then, concerning how to accommodate more papers at the conference,
> my preference also goes to solution A (i.e., parallel sessions).
>
> cheers
>
> Alain
>
>>> If this proposal were to be accepted, we would need to figure out  
>>> how
>>> to accommodate many more papers at the physical meeting.  How to
>>> achieve this is secondary to my main proposal, but a number of
>>> proposals have been floated, including
>>>
>>> * Parallel sessions
>>> * A lottery among accepted papers
>>> * Voting by conference registrants
>>> * Program committee decision
>>
>> I know this is secondary, but I want to make sure I get my two  
>> cents in: the only rational choice is to go to parallel sessions or  
>> to extend the length of the conference.
>>
>> I believe that voting, either by PC or conference registrants, has  
>> the potential to be much more unfair than current paper selection  
>> practice.  If part of the voting explicitly depends upon answering  
>> the question "who will give a good talk?" as opposed to "what is  
>> the content of the paper" then this introduces an extreme bias  
>> towards old, famous, successful researchers and away from young,  
>> new, unheard of researchers and students.  Whereas we now at least  
>> try to judge POPL papers purely on the merit of the current  
>> technical document, we would instead be veering away from that  
>> crucial principle.  And the more we start asking personality-based  
>> questions such as "who will give a good talk," the more we may be  
>> susceptible to subconscious biases against various minorities  
>> (women, racial, etc) or the more we may try to overcompensate for  
>> such biases, resulting in reverse-discrimination.
>>
>> I also believe that lottery for talks is bad.  What a lottery does  
>> is select some set of papers for which the talk audience is zero.   
>> With parallel sessions,
>> the talk audiences will be smaller, but not zero.  If I had a  
>> really great idea, I'd rather present it 6 months later at PLDI  
>> than have it appear 6 months earlier in the POPL proceedings, but  
>> not have the chance to give a talk.
>>
>> One last thing:  while we may be getting all tied in knots over  
>> this popl review process right now, from what I've heard, within  
>> computer science, our community is really pretty great when it  
>> comes to selecting papers for inclusion in conferences based on  
>> their merits.  I've heard of all kinds of dysfunctionality and  
>> biases and turf wars and sketchiness in other communities that we  
>> don't seem to be suffering from at all.  Of course, that's probably  
>> because we're constantly working to try to make the process better  
>> and more fair to all.
>>
>> Cheers,
>> Dave
>
>
> -- 
> -------------
> Alain GIRAULT                       http://pop-art.inrialpes.fr/~girault
> INRIA senior researcher             tel: +(33|0) 476 61 53 51
> Head of the POP ART project-team    fax: +(33|0) 476 61 52 52
> -----------------------------------------------------------------------------
> Sauvons la Recherche ! http://www.sauvonslarecherche.fr
> -----------------------------------------------------------------------------