This article presents a non-parametric statistical test that is very interesting for those who want to compare different heuristic algorithms that do not necessarily end with feasible (or satisfying) solutions. This test has been specially designed for working with very small sample sizes, meaning that a substantial computational effort can be saved when conducting numerical experiments. When the sample sizes are lower than 15, standard statistical tests for comparing the success rates of two populations cannot be validly used. So it is for very high confidence rates, even if sample sizes are larger than 15. Therefore, a non parametric test has been developed. This test is more powerful than Mc Nemar's one and can be applied for any sample sizes, but it requires relatively heavy computations. So, pre-computed values for 95% and 99% confidence levels have been tabulated in the article. The computation of confidence levels can also be done online at the URL: http://ina.eivd.ch/projects/stamp/
Citation
Metaheuristic international conference MIC'03, Kyoto, Japan, August, 2003