EUVIMEDDescubrir EUVIMED
Open access

Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics

Luning Sun, Christopher Gibbons, José Hernández-Orallo, Xiting Wang, Liming Jiang, David Stillwell, Fang Luo, Xing Xie

Journal of Medical Internet Research · 2025-05-26 · 27 · e70901-e70901

DOI: 10.2196/70901 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2025-05-26. Written by Luning Sun and 7 co-authors. Institution: The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom. Major MEDLINE topics: Artificial Intelligence, Psychometrics, Benchmarking. Cited 6 times so far (Crossref); 29 references. The full text is freely available in PubMed Central.

ResumenSubjectsDatos bibliográficosCiteRelated studies

Abstract (original language)

Rigorous evaluation of generalist medical artificial intelligence (GMAI) is imperative to ensure their utility and safety before implementation in health care. Current evaluation strategies rely heavily on benchmarks, which can suffer from issues with data contamination and cannot explain how GMAI might fail (lacking explanatory power) or in what circumstances (lacking predictive power). To address these limitations, we propose a new methodology to improve the quality of GMAI evaluation using construct-oriented processes. Drawing on modern psychometric techniques, we introduce approaches to construct identification and present alternative assessment formats for different domains of professional skills, knowledge, and behaviors that are essential for safe practice. We also discuss the need for human oversight in future GMAI adoption.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligencePsychometricsBenchmarkingHumans

Author keywords

benchmarkconstruct-oriented evaluationdata contaminationexplanatory powergeneralist medical artificial intelligencehealth carehuman oversightpredictive powerpsychometrics

Authors and institutions

  • The Psychometrics Centre, Cambridge Judge Business School, University of Cambridge, Cambridge, United Kingdom.
  • Oracle Health, Austin, TX, United States.
  • Valencian Research Institute for Artificial Intelligence (VRAIN), Universitat Politècnica de València, València, Spain.
  • Valencian Graduate School and Research Network of AI, València, Spain.
  • Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China.
  • Faculty of Psychology, Beijing Normal University, 19 Xinwai Ave, Beijing, 100875, China, 86 15120098365.

Cite

APA

Sun, L., Gibbons, C., Hernández-Orallo, J., Wang, X., Jiang, L., Stillwell, D., Luo, F., & Xie, X. (2025). Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics. Journal of Medical Internet Research, 27, e70901-e70901. https://doi.org/10.2196/70901

Vancouver

Sun L, Gibbons C, Hernández-Orallo J, Wang X, Jiang L, Stillwell D, et al. Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics. Journal of Medical Internet Research. 2025;27:e70901-e70901. doi:10.2196/70901

BibTeX
@article{Sun2025,
  author = {Sun, Luning and Gibbons, Christopher and Hernández-Orallo, José and Wang, Xiting and Jiang, Liming and Stillwell, David and Luo, Fang and Xie, Xing},
  title = {Beyond Benchmarks: Evaluating Generalist Medical Artificial Intelligence With Psychometrics},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e70901--e70901},
  doi = {10.2196/70901}
}
Start for free →