Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study
Arthritis & Rheumatology · 2024
DOI: 10.1002/art.42737 ↗Kurz gefasst
Fachartikel in der Zeitschrift Arthritis & Rheumatology, 2024. Verfasst von Carrie Ye und 4 weiteren Autor:innen. Beteiligte Einrichtung: University of Alberta, Edmonton, Alberta, Canada. Hauptthemen laut MEDLINE: Rheumatology, Physicians. Die Arbeit wurde bisher 77-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).
Abstract (Originalsprache)
Objective
The objective of the current study was to assess the quality of large language model (LLM) chatbot versus physician‐generated responses to patient‐generated rheumatology questions.
Methods
We conducted a single‐center cross‐sectional survey of rheumatology patients (n = 17) in Edmonton, Alberta, Canada. Patients evaluated LLM chatbot versus physician‐generated responses for comprehensiveness and readability, with four rheumatologists also evaluating accuracy by using a Likert scale from 1 to 10 (1 being poor, 10 being excellent).
Results
Patients rated no significant difference between artificial intelligence (AI) and physician‐generated responses in comprehensiveness (mean 7.12 ± SD 0.99 vs 7.52 ± 1.16; P = 0.1962) or readability (7.90 ± 0.90 vs 7.80 ± 0.75; P = 0.5905). Rheumatologists rated AI responses significantly poorer than physician responses on comprehensiveness (AI 5.52 ± 2.13 vs physician 8.76 ± 1.07; P < 0.0001), readability (AI 7.85 ± 0.92 vs physician 8.75 ± 0.57; P = 0.0003), and accuracy (AI 6.48 ± 2.07 vs physician 9.08 ± 0.64; P < 0.0001). The proportion of preference to AI‐ versus physician‐generated responses by patients and physicians was 0.45 ± 0.18 and 0.15 ± 0.08, respectively ( P = 0.0106). After learning that one answer for each question was AI generated, patients were able to correctly identify AI‐generated answers at a lower proportion compared to physicians (0.49 ± 0.26 vs 0.97 ± 0.04; P = 0.0183). The average word count of AI answers was 69.10 ± 25.35 words, as compared to 98.83 ± 34.58 words for physician‐generated responses ( P = 0.0008).
Conclusion
Rheumatology patients rated AI‐generated responses to patient questions similarly to physician‐generated responses in terms of comprehensiveness, readability, and overall preference. However, rheumatologists rated AI responses significantly poorer than physician‐generated responses, suggesting that LLM chatbot responses are inferior to physician responses, a difference that patients may not be aware of. image
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Autor:innen und Institutionen
- University of Alberta, Edmonton, Alberta, Canada.
- University Hospital Düsseldorf, Düsseldorf, Germany.
ORCID-Profile: Carrie Ye, Elric Zweck, Zechen Ma, Justin Smith, Steven Katz
Zitieren
APA
Ye, C., Zweck, E., Ma, Z., Smith, J., & Katz, S. (2024). Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study. Arthritis & Rheumatology. https://doi.org/10.1002/art.42737
Vancouver
Ye C, Zweck E, Ma Z, Smith J, Katz S. Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study. Arthritis & Rheumatology. 2024. doi:10.1002/art.42737
BibTeX
@article{Ye2024,
author = {Ye, Carrie and Zweck, Elric and Ma, Zechen and Smith, Justin and Katz, Steven},
title = {Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study},
journal = {Arthritis & Rheumatology},
year = {2024},
doi = {10.1002/art.42737}
}
EUVIMED