The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis
Die Genauigkeit und Leistungsfähigkeit von KI-Lösungen in Gesundheitsprüfungen und -zertifikaten: systematische Übersichtsarbeit und Metaanalyse
Journal of Medical Internet Research · 05.11.2024 · 26 · e56532
DOI: 10.2196/56532 ↗Auf einen Blick
LLMs erreichten in Gesundheitsprüfungen eine Gesamtgenauigkeit von 0,61. Für den USMLE lag die Genauigkeit bei 0,51; für ChatGPT bei 0,64.
Fragestellung
Wie genau sind LLMs in Gesundheitsprüfungen im Vergleich zu bekannten menschlichen Leistungsstandards?
Methode
Systematische Übersichtsarbeit und Metaanalyse; Suche nach Arbeiten bis 10. September 2023; Extraktion von Sensitivität, Genauigkeit und Präzision samt CIs.
Ergebnisse
1673 Zitate identifiziert; 1268 Titel/Abstracts gescreent; 32 Studien im Full-Text-Review. Metaanalyse: medizinische Prüfungsgenauigkeit 0,61 (CI 0,58–0,64); USMLE 0,51 (CI 0,46–0,56); ChatGPT 0,64 (CI 0,60–0,67).
Schlussfolgerung der Autor:innen
LLMs erscheinen vielversprechend, um den Bedarf im Gesundheitswesen zu unterstützen. Für Policy- und Implementierungsentscheidungen schlugen die Autor:innen das Rahmenwerk RUBRICC vor.
Kernaussagen
- Gesamtgenauigkeit von LLMs in medizinischen Prüfungen: 0,61
- USMLE-Genauigkeit: 0,51
- ChatGPT-Genauigkeit: 0,64
Deutsche Zusammenfassung: KI-gestützt aus dem Original-Abstract erstellt; alle Zahlen wurden automatisch mit dem Original abgeglichen. Maßgeblich ist der Original-Abstract.
Kurz gefasst
Metaanalyse in der Zeitschrift Journal of Medical Internet Research, online erschienen am 05.11.2024. Verfasst von William J Waldock und 5 weiteren Autor:innen. Beteiligte Einrichtung: Imperial College London, London, United Kingdom. Hauptthemen laut MEDLINE: Artificial Intelligence. Die Arbeit wurde bisher 53-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: Wellcome Trust.
Abstract (Originalsprache)
Background
Large language models (LLMs) have dominated public interest due to their apparent capability to accurately replicate learned knowledge in narrative text. However, there is a lack of clarity about the accuracy and capability standards of LLMs in health care examinations.
Objective
We conducted a systematic review of LLM accuracy, as tested under health care examination conditions, as compared to known human performance standards.
Methods
We quantified the accuracy of LLMs in responding to health care examination questions and evaluated the consistency and quality of study reporting. The search included all papers up until September 10, 2023, with all LLMs published in English journals that report clear LLM accuracy standards. The exclusion criteria were as follows: the assessment was not a health care exam, there was no LLM, there was no evaluation of comparable success accuracy, and the literature was not original research.The literature search included the following Medical Subject Headings (MeSH) terms used in all possible combinations: “artificial intelligence,” “ChatGPT,” “GPT,” “LLM,” “large language model,” “machine learning,” “neural network,” “Generative Pre-trained Transformer,” “Generative Transformer,” “Generative Language Model,” “Generative Model,” “medical exam,” “healthcare exam,” and “clinical exam.” Sensitivity, accuracy, and precision data were extracted, including relevant CIs.
Results
The search identified 1673 relevant citations. After removing duplicate results, 1268 (75.8%) papers were screened for titles and abstracts, and 32 (2.5%) studies were included for full-text review. Our meta-analysis suggested that LLMs are able to perform with an overall medical examination accuracy of 0.61 (CI 0.58-0.64) and a United States Medical Licensing Examination (USMLE) accuracy of 0.51 (CI 0.46-0.56), while Chat Generative Pretrained Transformer (ChatGPT) can perform with an overall medical examination accuracy of 0.64 (CI 0.6-0.67).
Conclusions
LLMs offer promise to remediate health care demand and staffing challenges by providing accurate and efficient context-specific information to critical decision makers. For policy and deployment decisions about LLMs to advance health care, we proposed a new framework called RUBRICC (Regulatory, Usability, Bias, Reliability [Evidence and Safety], Interoperability, Cost, and Codesign–Patient and Public Involvement and Engagement [PPIE]). This presents a valuable opportunity to direct the clinical commissioning of new LLM capabilities into health services, while respecting patient safety considerations. Trial Registration OSF Registries osf.io/xqzkw; https://osf.io/xqzkw
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Imperial College London, London, United Kingdom.
- Institute of Global Health Innovation, Imperial College London, London, United Kingdom.
ORCID-Profile: William J Waldock, Joe Zhang, Ahmad Guni, Ahmad Nabeel, Ara Darzi, Hutan Ashrafian
Förderung
- Wellcome Trust
Zitieren
APA
Waldock, W. J., Zhang, J., Guni, A., Nabeel, A., Darzi, A., & Ashrafian, H. (2024). The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis. Journal of Medical Internet Research, 26, e56532. https://doi.org/10.2196/56532
Vancouver
Waldock WJ, Zhang J, Guni A, Nabeel A, Darzi A, Ashrafian H. The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis. Journal of Medical Internet Research. 2024;26:e56532. doi:10.2196/56532
BibTeX
@article{Waldock2024,
author = {Waldock, William J and Zhang, Joe and Guni, Ahmad and Nabeel, Ahmad and Darzi, Ara and Ashrafian, Hutan},
title = {The Accuracy and Capability of Artificial Intelligence Solutions in Health Care Examinations and Certificates: Systematic Review and Meta-Analysis},
journal = {Journal of Medical Internet Research},
year = {2024},
volume = {26},
pages = {e56532},
doi = {10.2196/56532}
}
EUVIMED