EUVIMEDEUVIMED entdecken
Open Access

Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics

Luning Sun, Christopher Gibbons, José Hernández-Orallo, Xiting Wang, Liming Jiang, David Stillwell, Fang Luo, Xing Xie

Journal of Medical Internet Research · 26.05.2025 · 27 · e70901-e70901

DOI: 10.2196/70901 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 26.05.2025. Verfasst von Luning Sun und 7 weiteren Autor:innen. Beteiligte Einrichtung: The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom. Hauptthemen laut MEDLINE: Artificial Intelligence, Psychometrics, Benchmarking. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 29 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligencePsychometricsBenchmarkingHumans

Stichwörter der Autor:innen

benchmarkconstruct-oriented evaluationdata contaminationexplanatory powergeneralist medical artificial intelligencehealth carehuman oversightpredictive powerpsychometrics

Autor:innen und Institutionen

  • The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom.
  • Oracle Health, Austin, TX, United States.
  • Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, València, Spain.
  • Valencian Graduate School and Research Network of AI, València, Spain.
  • Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
  • Faculty of Psychology, Beijing Normal University, 19 Xinwai Ave, Beijing, 100875, China, 86 15120098365.

Zitieren

APA

Sun, L., Gibbons, C., Hernández-Orallo, J., Wang, X., Jiang, L., Stillwell, D., Luo, F., & Xie, X. (2025). Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics. Journal of Medical Internet Research, 27, e70901-e70901. https://doi.org/10.2196/70901

Vancouver

Sun L, Gibbons C, Hernández-Orallo J, Wang X, Jiang L, Stillwell D, et al. Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics. Journal of Medical Internet Research. 2025;27:e70901-e70901. doi:10.2196/70901

BibTeX
@article{Sun2025,
  author = {Sun, Luning and Gibbons, Christopher and Hernández-Orallo, José and Wang, Xiting and Jiang, Liming and Stillwell, David and Luo, Fang and Xie, Xing},
  title = {Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e70901--e70901},
  doi = {10.2196/70901}
}
Kostenlos starten →