[Icfp04-discuss] Method for determining the best ant

[Matt Kearse]

>> The point I find wrong is repeating the tournament with a small
>> sub-set of the original submissions. These 2 methods can easily result
>> in different winning ants at each stage.

> We have considered such possibilities very well and let the task
> description intentionally ambiguous, so that we have the freedom to
> choose a method which is physically doable and as reasonable as
> possible to everybody.

> Don't worry, please - we observe that the 2 methods are _very_
> unlikely to give different conclusions anyway.  I.e., whatever
> reasonable methods we choose, the winner(s) would not change in fact.

I decided to do some simulations of this and found out the 2 methods
to give quite different results. I ran all the ants in the zip file
someone posted against all others on all the sample maps many times. I
then ran the top 10 ants from the first simulation against only each
other in a second simulation. Since the organizers don't want the
results made public, I have replaced the team names with letters.
Scores are the number of points obtained (2 for a win, 1 for a draw)

Top 10 after playing all ants (about 70 different ants) against all
ants on each sample map 10 times (5 as each ant color)

A-1 13200
A-2 13083
B-1 12309
C-1 11932
D-1 11887
E-1 11817
F-1 11814
G-1 11807
E-2 11783
C-2 11740

Clearly team A is the winner, although it is fairly close between
their 2 ant variants.

Now playing only those top 10 ants against each other:

A-2 11138
A-1 10670
D-1 7383
B-1 6521
C-1 4971
F-1 4811
E-2 4678
E-1 4628
C-2 4296
G-1 3904

Team A is still clearly the winner, although their other ant variant
wins this time. However, more interesting is that the team in 2nd
place (ant in 3rd place) now goes to the team that came in 4th place
(ant in 5th place) in the original competition. If Team A hadn't
entered, then it is quite likely the winner of this tournament would
depend on whether the judges decided if there was a "clear winner" in
the first round or not (assuming they chose 10 ants for the finalists
tournament - of course the number of ants they chose for the finalists
also makes a big difference to the results)

In my opinion, the reasons for the differences comes down to the fact
that some ants are better at fighting other good ants, but others are
better at taking advantage of the large pool of not so good ants. Team
A are obviously very good at both, and even seem to have an ant
optimized for each case.

I also wonder if the organizers consider the case of an unclear winner
between 2 ants from a single team to be reason to go to the second
method of judging. Their decision could make quite a difference to who
comes in second place in the competition.

Note: Of course this is all run with a sample of about 70 ants
submitted, so actual competition may (and probably will) differ.

Matt.