EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

What Should an AI Assessor Optimise for?

Daniel Romero-Alvarado, Fernando Martínez-Plumed, José Hernández-Orallo

Machine Learning · 2026-07 · 115 (7)

DOI: 10.1007/s10994-026-07084-0 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Machine Learning, online erschienen am 02.07.2026. Verfasst von Daniel Romero-Alvarado und 2 weiteren Autor:innen. Die Arbeit nennt 66 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Universitat Politècnica de València.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

An AI assessor is an external, ideally independent system that predicts an indicator, e.g., a loss value, of another AI system. Assessors can leverage information from the test results of many other AI systems and have the flexibility of being trained on any loss function or scoring rule: from squared error to toxicity metrics. Here we address the question: is it always optimal to train the assessor for the target metric? Or could it be better to train for a different metric and then map predictions back to the target metric? Using twenty regression and classification problems with tabular data, we experimentally explore this question for, respectively, regression losses and classification scores with monotonic and nonmonotonic mappings and find that, contrary to intuition, optimising for more informative metrics (i.e., yielding a better-conditioned supervision signal) is not universally preferred. Surprisingly, some monotonic transformations are promising. For example, logistic loss is useful for minimising absolute or quadratic errors in regression, and logarithmic score helps maximise quadratic or spherical scores in classification.

Förderung

  • Universitat Politècnica de València

Zitieren

APA

Romero-Alvarado, D., Martínez-Plumed, F., & Hernández-Orallo, J. (2026). What Should an AI Assessor Optimise for?. Machine Learning, 115(7). https://doi.org/10.1007/s10994-026-07084-0

Vancouver

Romero-Alvarado D, Martínez-Plumed F, Hernández-Orallo J. What Should an AI Assessor Optimise for?. Machine Learning. 2026;115(7). doi:10.1007/s10994-026-07084-0

BibTeX
@article{RomeroAlvarado2026,
  author = {Romero-Alvarado, Daniel and Martínez-Plumed, Fernando and Hernández-Orallo, José},
  title = {What Should an AI Assessor Optimise for?},
  journal = {Machine Learning},
  year = {2026},
  volume = {115},
  number = {7},
  doi = {10.1007/s10994-026-07084-0}
}
Kostenlos starten →