Some Common Mistakes In IR Evaluation, And How They Can Be Avoided
ACM SIGIR Forum · 22.02.2018 · 51 (3) · 32-41
DOI: 10.1145/3190580.3190586 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM SIGIR Forum, online erschienen am 22.02.2018. Verfasst von Norbert Fuhr. Die Arbeit wurde bisher 105-mal zitiert (Crossref) und nennt 22 Literaturangaben.
Abstract (Originalsprache)
This paper points out some mistakes that can be frequently found in IR publications: MRR and ERR violate basic requirements for a metric, MAP is based on unrealistic assumptions, the numbers shown overstate the precision of the result, relative improvements of arithmetic means are inappropriate, the simple holdout method yields unreliable results, hypotheses are often formulated after the experiment, significance tests frequently ignore the multiple comparisons problem, effect sizes are ignored, reproducibility of the experiments might be nearly impossible, and sometimes authors claim proof by experimentation.
Zitieren
APA
Fuhr, N. (2018). Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. ACM SIGIR Forum, 51(3), 32-41. https://doi.org/10.1145/3190580.3190586
Vancouver
Fuhr N. Some Common Mistakes In IR Evaluation, And How They Can Be Avoided. ACM SIGIR Forum. 2018;51(3):32-41. doi:10.1145/3190580.3190586
BibTeX
@article{Fuhr2018,
author = {Fuhr, Norbert},
title = {Some Common Mistakes In IR Evaluation, And How They Can Be Avoided},
journal = {ACM SIGIR Forum},
year = {2018},
volume = {51},
number = {3},
pages = {32--41},
doi = {10.1145/3190580.3190586}
}
EUVIMED