[Icfp04-discuss] Method for determining the best ant

[Matt Kearse]

I think the method the judges use to determine the winners to be not
well defined:

>During the tournament, each pair of submissions is pitted against
>each other twice on each of the contest worlds---once with the first
>submission playing red and the second black, and once with the first
>playing black and the second red. A submission gains 2 points for
>each game it wins, and 1 point for each draw. The submission with the
>most points wins the tournament. The number of the worlds used during
>the tournament is unspecified, but will be large enough for
>determining a clear winner. If there is nevertheless no clear winner,
>the tournament is repeated with a certain number of finalist
>submissions. The seed used by the random number generator is
>unspecified.

The point I find wrong is repeating the tournament with a small
sub-set of the original submissions. These 2 methods can easily result
in different winning ants at each stage.

The reason for this comes from the fact that some of the worlds are
grossly biased in favour of 1 color ant - lets say black as is the
case in sample world 9. When good ants play against other good ants on
such maps, then they will each almost always win when playing as
black. But when played against a bad ant, a good ant can sometimes win
even when playing as red. But ants can be optimized for one of 2
things -

1) Beating another good ant when playing on a fairly even map.
2) Beating bad ants when playing at a disadvantage on an uneven map.

It seems that the judges will essentially be testing for condition 2,
but if there is no clear winner (how far ahead does an ant have to be
do be declared a "clear winner"?), then they will switch to using
method 1 since they play with just a small subset of the total
submissions.

For example, from a few tests I have run I find my ant will loose more
often that not to an ant like red teams ant. But when I add a third
party ant and play us both against them, my ant can often win when at
a disadvantage whereas red teams would not. Overall my ant could win a
higher number of games in total but for just the sub-set of my ant
versus red team's ant, my ant would come out worse. I haven't done
extensive testing with other ants so I have no idea which ants will
actually perform best under tournament conditions, and really, none of
us can know without access to all the ants including the bad ones.

So determining the winning ant may actually comes down to what the
judges define as a "clear winner" in the first stage, and the number
of finalists they choose in the second stage of judging if any. And of
course the quality and quantity of all ants in the competition,
including the bad ones.

In my opinion, the judges should have chosen to either:
1) Only play all ants versus all ants and not have a second round of
consisting of just the finalists. They can keep playing more rounds
until one ant does eventually win.
or
2) Play all versus all, removing the worst performing ant at each
stage and repeating until it comes down to final of one on one.

Both methods are valid for judging, but they test for different
things. Anyone else have opinions on this matter?

Matt.