[POPLmark] The purpose of challenge problems?

[Karl Crary]

It's struck me that there's been a bit of a disconnect in the last 
week's discussion of new challenge problems.  I think the issue is that 
we are not in agreement as to their purpose.  So I'd like to ask the 
question, what is the purpose to a new challenge problem?

The purpose of the original Poplmark challenge, as I understood it, was 
to pose a problem dealing with an interesting set of language features 
in order to see how well various technologies dealt with those 
features.  Moreover, in order to keep the challenge from being too 
artificial, the problem was selected to be one with some intrinsic merit.

So what would be the purpose of a new set of smaller, easier problems?  
I'm not sure.  Perhaps they are just for fun, in which case I'm simply 
being a killjoy.  But I don't see any way in which they advance our 
state of knowledge about the area.

There's also been talk of proposing another Poplmark-sized challenge, 
exercising a different set of issues.  How profitable that would depends 
on how well the selection of features is made.  But it seems clear in 
any case that each iteration tells us less.

So I would like to suggest an alternative challenge problem: Use the 
system of your choice to formalize your own work.  This challenge is 
natural, and is guaranteed to exercise the features of interest to you.

Clearly this is the future of the area, so is there any reason why we 
shouldn't get on with it?  Challenge problems are fun, but I would much 
rather read about real results.

    -- Karl