Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics
Journal of Medical Internet Research · 2025-05-26 · 27 · e70901-e70901
DOI: 10.2196/70901 ↗At a glance
Journal article in Journal of Medical Internet Research, published online on 2025-05-26. Written by Luning Sun and 7 co-authors. Institution: The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom. Major MEDLINE topics: Artificial Intelligence, Psychometrics, Benchmarking. Cited 6 times so far (Crossref); 29 references. The full text is freely available in PubMed Central.
Abstract (original language)
Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom.
- Oracle Health, Austin, TX, United States.
- Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, València, Spain.
- Valencian Graduate School and Research Network of AI, València, Spain.
- Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
- Faculty of Psychology, Beijing Normal University, 19 Xinwai Ave, Beijing, 100875, China, 86 15120098365.
Cite
APA
Sun, L., Gibbons, C., Hernández-Orallo, J., Wang, X., Jiang, L., Stillwell, D., Luo, F., & Xie, X. (2025). Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics. Journal of Medical Internet Research, 27, e70901-e70901. https://doi.org/10.2196/70901
Vancouver
Sun L, Gibbons C, Hernández-Orallo J, Wang X, Jiang L, Stillwell D, et al. Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics. Journal of Medical Internet Research. 2025;27:e70901-e70901. doi:10.2196/70901
BibTeX
@article{Sun2025,
author = {Sun, Luning and Gibbons, Christopher and Hernández-Orallo, José and Wang, Xiting and Jiang, Liming and Stillwell, David and Luo, Fang and Xie, Xing},
title = {Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e70901--e70901},
doi = {10.2196/70901}
}
EUVIMED