EUVIMEDEUVIMED entdecken
Open Access

Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions

Amulya Yalamanchili, Bishwambhar Sengupta, Joshua Song, Sara Lim, Tarita O Thomas, Bharat B Mittal, Mohamed E Abazeed, P Troy Teo

JAMA Network Open · 02.04.2024 · 7 (4) · e244630

DOI: 10.1001/jamanetworkopen.2024.4630 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JAMA Network Open, online erschienen am 02.04.2024. Verfasst von Amulya Yalamanchili und 7 weiteren Autor:innen. Beteiligte Einrichtung: Robert H. Lurie Comprehensive Cancer Center, Department of Radiation Oncology, Northwestern Memorial Hospital, Northwestern University Feinberg Schoo…. Hauptthemen laut MEDLINE: Radiation Oncology. Die Arbeit wurde bisher 138-mal zitiert (Crossref) und nennt 36 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NCI NIH HHS, CIHR.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Importance Artificial intelligence (AI) large language models (LLMs) demonstrate potential in simulating human-like dialogue. Their efficacy in accurate patient-clinician communication within radiation oncology has yet to be explored.

Objective

To determine an LLM’s quality of responses to radiation oncology patient care questions using both domain-specific expertise and domain-agnostic metrics. Design, Setting, and Participants This cross-sectional study retrieved questions and answers from websites (accessed February 1 to March 20, 2023) affiliated with the National Cancer Institute and the Radiological Society of North America. These questions were used as queries for an AI LLM, ChatGPT version 3.5 (accessed February 20 to April 20, 2023), to prompt LLM-generated responses. Three radiation oncologists and 3 radiation physicists ranked the LLM-generated responses for relative factual correctness, relative completeness, and relative conciseness compared with online expert answers. Statistical analysis was performed from July to October 2023. Main Outcomes and Measures The LLM’s responses were ranked by experts using domain-specific metrics such as relative correctness, conciseness, completeness, and potential harm compared with online expert answers on a 5-point Likert scale. Domain-agnostic metrics encompassing cosine similarity scores, readability scores, word count, lexicon, and syllable counts were computed as independent quality checks for LLM-generated responses.

Results

Of the 115 radiation oncology questions retrieved from 4 professional society websites, the LLM performed the same or better in 108 responses (94%) for relative correctness, 89 responses (77%) for completeness, and 105 responses (91%) for conciseness compared with expert answers. Only 2 LLM responses were ranked as having potential harm. The mean (SD) readability consensus score for expert answers was 10.63 (3.17) vs 13.64 (2.22) for LLM answers ( P < .001), indicating 10th grade and college reading levels, respectively. The mean (SD) number of syllables was 327.35 (277.15) for expert vs 376.21 (107.89) for LLM answers ( P = .07), the mean (SD) word count was 226.33 (191.92) for expert vs 246.26 (69.36) for LLM answers ( P = .27), and the mean (SD) lexicon score was 200.15 (171.28) for expert vs 219.10 (61.59) for LLM answers ( P = .24). Conclusions and Relevance In this cross-sectional study, the LLM generated accurate, comprehensive, and concise responses with minimal risk of harm, using language similar to human experts but at a higher reading level. These findings suggest the LLM’s potential, with some retraining, as a valuable resource for patient queries in radiation oncology and other medical fields.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Radiation OncologyHumansArtificial IntelligenceCross-Sectional StudiesLanguagePatient Care

Autor:innen und Institutionen

  • Robert H. Lurie Comprehensive Cancer Center, Department of Radiation Oncology, Northwestern Memorial Hospital, Northwestern University Feinberg School of Medicine, Chicago, Illinois.

Förderung

  • NCI NIH HHS
  • CIHR

Zitieren

APA

Yalamanchili, A., Sengupta, B., Song, J., Lim, S., Thomas, T. O., Mittal, B. B., Abazeed, M. E., & Teo, P. T. (2024). Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions. JAMA Network Open, 7(4), e244630. https://doi.org/10.1001/jamanetworkopen.2024.4630

Vancouver

Yalamanchili A, Sengupta B, Song J, Lim S, Thomas TO, Mittal BB, et al. Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions. JAMA Network Open. 2024;7(4):e244630. doi:10.1001/jamanetworkopen.2024.4630

BibTeX
@article{Yalamanchili2024,
  author = {Yalamanchili, Amulya and Sengupta, Bishwambhar and Song, Joshua and Lim, Sara and Thomas, Tarita O and Mittal, Bharat B and Abazeed, Mohamed E and Teo, P Troy},
  title = {Quality of Large Language Model Responses to Radiation Oncology Patient Care Questions},
  journal = {JAMA Network Open},
  year = {2024},
  volume = {7},
  number = {4},
  pages = {e244630},
  doi = {10.1001/jamanetworkopen.2024.4630}
}
Kostenlos starten →