EUVIMEDEUVIMED entdecken
QuerschnittstudieOpen Access · CC BY 4.0

Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study

Arun James Thirunavukarasu iD, Shathar Mahmood, Andrew Malem, William Paul Foster, Rohan Sanghera iD, Refaat Hassan, Sean Zhou, Shiao Wei Wong, Yee Ling Wong, Yu Jeat Chong, Abdullah Shakeel, Yin-Hsi Chang und 6 weitere

PLOS Digital Health · 17.04.2024 · 3 (4) · e0000341

DOI: 10.1371/journal.pdig.0000341 ↗

Kurz gefasst

Querschnittstudie in der Zeitschrift PLOS Digital Health, online erschienen am 17.04.2024. Verfasst von Arun James Thirunavukarasu und 17 weiteren Autor:innen. Beteiligte Einrichtung: University of Cambridge School of Clinical Medicine, Cambridge, United Kingdom. Die Arbeit wurde bisher 95-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Medical Research Council, Singapore, National Research Foundation Singapore, Duke-NUS Medical School.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) underlie remarkable recent advanced in natural language processing, and they are beginning to be applied in clinical contexts. We aimed to evaluate the clinical potential of state-of-the-art LLMs in ophthalmology using a more robust benchmark than raw examination scores. We trialled GPT-3.5 and GPT-4 on 347 ophthalmology questions before GPT-3.5, GPT-4, PaLM 2, LLaMA, expert ophthalmologists, and doctors in training were trialled on a mock examination of 87 questions. Performance was analysed with respect to question subject and type (first order recall and higher order reasoning). Masked ophthalmologists graded the accuracy, relevance, and overall preference of GPT-3.5 and GPT-4 responses to the same questions. The performance of GPT-4 (69%) was superior to GPT-3.5 (48%), LLaMA (32%), and PaLM 2 (56%). GPT-4 compared favourably with expert ophthalmologists (median 76%, range 64–90%), ophthalmology trainees (median 59%, range 57–63%), and unspecialised junior doctors (median 43%, range 41–44%). Low agreement between LLMs and doctors reflected idiosyncratic differences in knowledge and reasoning with overall consistency across subjects and types ( p >0.05). All ophthalmologists preferred GPT-4 responses over GPT-3.5 and rated the accuracy and relevance of GPT-4 as higher ( p

Autor:innen und Institutionen

  • University of Cambridge School of Clinical Medicine, Cambridge, United Kingdom.
  • Oxford University Clinical Academic Graduate School, University of Oxford, Oxford, United Kingdom.
  • Eye Institute, Cleveland Clinic Abu Dhabi, Abu Dhabi Emirate, United Arab Emirates.
  • Department of Physiology, Development and Neuroscience, University of Cambridge, Cambridge, United Kingdom.
  • West Suffolk NHS Foundation Trust, Bury St Edmunds, United Kingdom.
  • Manchester Royal Eye Hospital, Manchester University NHS Foundation Trust, Manchester, United Kingdom.

ORCID-Profile: Arun James Thirunavukarasu, Rohan Sanghera, Saaeha Rauz

Förderung

  • National Medical Research Council, Singapore
  • National Research Foundation Singapore
  • Duke-NUS Medical School
  • Agency for Science, Technology and Research
  • Medical Research Council

Zitieren

APA

Thirunavukarasu, A. J., Mahmood, S., Malem, A., Foster, W. P., Sanghera, R., Hassan, R., Zhou, S., Wong, S. W., Wong, Y. L., Chong, Y. J., Shakeel, A., Chang, Y. H., Tan, B. K. J., Jain, N., Tan, T. F., Rauz, S., Ting, D. S. W., & Ting, D. S. J. (2024). Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study. PLOS Digital Health, 3(4), e0000341. https://doi.org/10.1371/journal.pdig.0000341

Vancouver

Thirunavukarasu AJ, Mahmood S, Malem A, Foster WP, Sanghera R, Hassan R, et al. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study. PLOS Digital Health. 2024;3(4):e0000341. doi:10.1371/journal.pdig.0000341

BibTeX
@article{Thirunavukarasu2024,
  author = {Thirunavukarasu, Arun James and Mahmood, Shathar and Malem, Andrew and Foster, William Paul and Sanghera, Rohan and Hassan, Refaat and Zhou, Sean and Wong, Shiao Wei and Wong, Yee Ling and Chong, Yu Jeat and Shakeel, Abdullah and Chang, Yin-Hsi and Tan, Benjamin Kye Jyn and Jain, Nikhil and Tan, Ting Fang and Rauz, Saaeha and Ting, Daniel Shu Wei and Ting, Darren Shu Jeng},
  title = {Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study},
  journal = {PLOS Digital Health},
  year = {2024},
  volume = {3},
  number = {4},
  pages = {e0000341},
  doi = {10.1371/journal.pdig.0000341}
}
Kostenlos starten →