EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study

Carrie Ye iD, Elric Zweck iD, Zechen Ma iD, Justin Smith iD, Steven Katz iD

Arthritis & Rheumatology · 2024

DOI: 10.1002/art.42737 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Arthritis & Rheumatology, 2024. Verfasst von Carrie Ye und 4 weiteren Autor:innen. Beteiligte Einrichtung: University of Alberta, Edmonton, Alberta, Canada. Hauptthemen laut MEDLINE: Rheumatology, Physicians. Die Arbeit wurde bisher 77-mal zitiert (Crossref) und nennt 15 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

The objective of the current study was to assess the quality of large language model (LLM) chatbot versus physician‐generated responses to patient‐generated rheumatology questions.

Methods

We conducted a single‐center cross‐sectional survey of rheumatology patients (n = 17) in Edmonton, Alberta, Canada. Patients evaluated LLM chatbot versus physician‐generated responses for comprehensiveness and readability, with four rheumatologists also evaluating accuracy by using a Likert scale from 1 to 10 (1 being poor, 10 being excellent).

Results

Patients rated no significant difference between artificial intelligence (AI) and physician‐generated responses in comprehensiveness (mean 7.12 ± SD 0.99 vs 7.52 ± 1.16; P = 0.1962) or readability (7.90 ± 0.90 vs 7.80 ± 0.75; P = 0.5905). Rheumatologists rated AI responses significantly poorer than physician responses on comprehensiveness (AI 5.52 ± 2.13 vs physician 8.76 ± 1.07; P < 0.0001), readability (AI 7.85 ± 0.92 vs physician 8.75 ± 0.57; P = 0.0003), and accuracy (AI 6.48 ± 2.07 vs physician 9.08 ± 0.64; P < 0.0001). The proportion of preference to AI‐ versus physician‐generated responses by patients and physicians was 0.45 ± 0.18 and 0.15 ± 0.08, respectively ( P = 0.0106). After learning that one answer for each question was AI generated, patients were able to correctly identify AI‐generated answers at a lower proportion compared to physicians (0.49 ± 0.26 vs 0.97 ± 0.04; P = 0.0183). The average word count of AI answers was 69.10 ± 25.35 words, as compared to 98.83 ± 34.58 words for physician‐generated responses ( P = 0.0008).

Conclusion

Rheumatology patients rated AI‐generated responses to patient questions similarly to physician‐generated responses in terms of comprehensiveness, readability, and overall preference. However, rheumatologists rated AI responses significantly poorer than physician‐generated responses, suggesting that LLM chatbot responses are inferior to physician responses, a difference that patients may not be aware of. image

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

RheumatologyPhysiciansHumansArtificial IntelligenceCross-Sectional StudiesAlbertaLanguage

Autor:innen und Institutionen

  • University of Alberta, Edmonton, Alberta, Canada.
  • University Hospital Düsseldorf, Düsseldorf, Germany.

ORCID-Profile: Carrie Ye, Elric Zweck, Zechen Ma, Justin Smith, Steven Katz

Zitieren

APA

Ye, C., Zweck, E., Ma, Z., Smith, J., & Katz, S. (2024). Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study. Arthritis & Rheumatology. https://doi.org/10.1002/art.42737

Vancouver

Ye C, Zweck E, Ma Z, Smith J, Katz S. Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study. Arthritis & Rheumatology. 2024. doi:10.1002/art.42737

BibTeX
@article{Ye2024,
  author = {Ye, Carrie and Zweck, Elric and Ma, Zechen and Smith, Justin and Katz, Steven},
  title = {Doctor Versus Artificial Intelligence: Patient and Physician Evaluation of Large Language Model Responses to Rheumatology Patient Questions in a Cross‐Sectional Study},
  journal = {Arthritis & Rheumatology},
  year = {2024},
  doi = {10.1002/art.42737}
}
Kostenlos starten →