EUVIMEDOtkrijte EUVIMED
MetaanalyseSystematische ÜbersichtOpen access

The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis

Die Genauigkeit und Leistungsfähigkeit von KI-Lösungen in Gesundheitsprüfungen und -zertifikaten: systematische Übersichtsarbeit und Metaanalyse

William J Waldock iD, Joe Zhang iD, Ahmad Guni iD, Ahmad Nabeel iD, Ara Darzi iD, Hutan Ashrafian iD

Journal of Medical Internet Research · 2024-11-05 · 26 · e56532

DOI: 10.2196/56532 ↗

Auf einen Blick

LLMs erreichten in Gesundheitsprüfungen eine Gesamtgenauigkeit von 0,61. Für den USMLE lag die Genauigkeit bei 0,51; für ChatGPT bei 0,64.

Fragestellung

Wie genau sind LLMs in Gesundheitsprüfungen im Vergleich zu bekannten menschlichen Leistungsstandards?

Methode

Systematische Übersichtsarbeit und Metaanalyse; Suche nach Arbeiten bis 10. September 2023; Extraktion von Sensitivität, Genauigkeit und Präzision samt CIs.

Ergebnisse

1673 Zitate identifiziert; 1268 Titel/Abstracts gescreent; 32 Studien im Full-Text-Review. Metaanalyse: medizinische Prüfungsgenauigkeit 0,61 (CI 0,58–0,64); USMLE 0,51 (CI 0,46–0,56); ChatGPT 0,64 (CI 0,60–0,67).

Schlussfolgerung der Autor:innen

LLMs erscheinen vielversprechend, um den Bedarf im Gesundheitswesen zu unterstützen. Für Policy- und Implementierungsentscheidungen schlugen die Autor:innen das Rahmenwerk RUBRICC vor.

Kernaussagen

  • Gesamtgenauigkeit von LLMs in medizinischen Prüfungen: 0,61
  • USMLE-Genauigkeit: 0,51
  • ChatGPT-Genauigkeit: 0,64

Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.

At a glance

Metaanalyse in Journal of Medical Internet Research, published online on 2024-11-05. Written by William J Waldock and 5 co-authors. Institution: Imperial College London, London, United Kingdom. Major MEDLINE topics: Artificial Intelligence. Cited 53 times so far (Crossref); 71 references. The full text is freely available in PubMed Central. Funded by: Wellcome Trust.

Auf einen BlickSažetakSubjectsBibliografski podaciCiteRelated studies

Abstract (original language)

Background

Large language models (LLMs) have dominated public interest due to their apparent capability to accurately replicate learned knowledge in narrative text. However, there is a lack of clarity about the accuracy and capability standards of LLMs in health care examinations.

Objective

We conducted a systematic review of LLM accuracy, as tested under health care examination conditions, as compared to known human performance standards.

Methods

We quantified the accuracy of LLMs in responding to health care examination questions and evaluated the consistency and quality of study reporting. The search included all papers up until September 10, 2023, with all LLMs published in English journals that report clear LLM accuracy standards. The exclusion criteria were as follows: the assessment was not a health care exam, there was no LLM, there was no evaluation of comparable success accuracy, and the literature was not original research.The literature search included the following Medical Subject Headings (MeSH) terms used in all possible combinations: “artificial intelligence,” “ChatGPT,” “GPT,” “LLM,” “large language model,” “machine learning,” “neural network,” “Generative Pre-trained Transformer,” “Generative Transformer,” “Generative Language Model,” “Generative Model,” “medical exam,” “healthcare exam,” and “clinical exam.” Sensitivity, accuracy, and precision data were extracted, including relevant CIs.

Results

The search identified 1673 relevant citations. After removing duplicate results, 1268 (75.8%) papers were screened for titles and abstracts, and 32 (2.5%) studies were included for full-text review. Our meta-analysis suggested that LLMs are able to perform with an overall medical examination accuracy of 0.61 (CI 0.58-0.64) and a United States Medical Licensing Examination (USMLE) accuracy of 0.51 (CI 0.46-0.56), while Chat Generative Pretrained Transformer (ChatGPT) can perform with an overall medical examination accuracy of 0.64 (CI 0.6-0.67).

Conclusions

LLMs offer promise to remediate health care demand and staffing challenges by providing accurate and efficient context-specific information to critical decision makers. For policy and deployment decisions about LLMs to advance health care, we proposed a new framework called RUBRICC (Regulatory, Usability, Bias, Reliability [Evidence and Safety], Interoperability, Cost, and Codesign–Patient and Public Involvement and Engagement [PPIE]). This presents a valuable opportunity to direct the clinical commissioning of new LLM capabilities into health services, while respecting patient safety considerations. Trial Registration OSF Registries osf.io/xqzkw; https://osf.io/xqzkw

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceHumansCertification

Author keywords

AILLMartificial intelligenceclinical commissioninghealth care examhealth care examinationhealth serviceslarge language modelnarrative medical responsesafety

Authors and institutions

  • Imperial College London, London, United Kingdom.
  • Institute of Global Health Innovation, Imperial College London, London, United Kingdom.

ORCID profiles: William J Waldock, Joe Zhang, Ahmad Guni, Ahmad Nabeel, Ara Darzi, Hutan Ashrafian

Funding

  • Wellcome Trust

Cite

APA

Waldock, W. J., Zhang, J., Guni, A., Nabeel, A., Darzi, A., & Ashrafian, H. (2024). The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis. Journal of Medical Internet Research, 26, e56532. https://doi.org/10.2196/56532

Vancouver

Waldock WJ, Zhang J, Guni A, Nabeel A, Darzi A, Ashrafian H. The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis. Journal of Medical Internet Research. 2024;26:e56532. doi:10.2196/56532

BibTeX
@article{Waldock2024,
  author = {Waldock, William J and Zhang, Joe and Guni, Ahmad and Nabeel, Ahmad and Darzi, Ara and Ashrafian, Hutan},
  title = {The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e56532},
  doi = {10.2196/56532}
}
Start for free →