Generative artificial intelligence as a source of breast cancer information for patients: Proceed with caution
Cancer · 2025
DOI: 10.1002/cncr.35521 ↗Kurz gefasst
Fachartikel in der Zeitschrift Cancer, 2025. Verfasst von Ko Un Park und 16 weiteren Autor:innen. Beteiligte Einrichtung: Division of Breast Surgery, Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA. Hauptthemen laut MEDLINE: Breast Neoplasms, Artificial Intelligence. Die Arbeit wurde bisher 16-mal zitiert (Crossref) und nennt 27 Literaturangaben.
Abstract (Originalsprache)
Background
This study evaluated the accuracy, clinical concordance, and readability of the chatbot interface generative pretrained transformer (ChatGPT) 3.5 as a source of breast cancer information for patients.
Methods
Twenty questions that patients are likely to ask ChatGPT were identified by breast cancer advocates. These were posed to ChatGPT 3.5 in July 2023 and were repeated three times. Responses were graded in two domains: accuracy (4‐point Likert scale, 4 = worst) and clinical concordance (information is clinically similar to physician response; 5‐point Likert scale, 5 = not similar at all). The concordance of responses with repetition was estimated using intraclass correlation coefficient (ICC) of word counts. Response readability was calculated using the Flesch Kincaid readability scale. References were requested and verified.
Results
The overall average accuracy was 1.88 (range 1.0–3.0; 95% confidence interval [CI], 1.42–1.94), and clinical concordance was 2.79 (range 1.0–5.0; 95% CI, 1.94–3.64). The average word count was 310 words per response (range, 146–441 words per response) with high concordance (ICC, 0.75; 95% CI, 0.59–0.91; p < .001). The average readability was poor at 37.9 (range, 18.0–60.5) with high concordance (ICC, 0.73; 95% CI, 0.57–0.90; p < .001). There was a weak correlation between ease of readability and better clinical concordance (−0.15; p = .025). Accuracy did not correlate with readability (0.05; p = .079). The average number of references was 1.97 (range, 1–4; total, 119). ChatGPT cited peer‐reviewed articles only once and often referenced nonexistent websites (41%).
Conclusions
Because ChatGPT 3.5 responses were incorrect 24% of the time and did not provide real references 41% of the time, patients should be cautioned about using ChatGPT for medical information.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Division of Breast Surgery, Department of Surgery, Brigham and Women's Hospital, Boston, Massachusetts, USA.
- Breast Oncology Program, Dana-Farber Brigham Cancer Center, Boston, Massachusetts, USA.
- Harvard Medical School, Boston, Massachusetts, USA.
- Center for Surgery and Public Health, Brigham and Women's Hospital, Boston, Massachusetts, USA.
- Medical Oncology, Dana-Farber Cancer Institute, Boston, Massachusetts, USA.
- Radiation Oncology, Dana-Farber Brigham Cancer Center, Boston, Massachusetts, USA.
ORCID-Profile: Ko Un Park, Filipa Lynce, Tari A King
Zitieren
APA
Park, K. U., Lipsitz, S., Dominici, L. S., Lynce, F., Minami, C. A., Nakhlis, F., Waks, A. G., Warren, L. E., Eidman, N., Frazier, J., Hernandez, L., Leslie, C., Rafte, S., Stroud, D., Weissman, J. S., King, T. A., & Mittendorf, E. A. (2025). Generative artificial intelligence as a source of breast cancer information for patients: Proceed with caution. Cancer. https://doi.org/10.1002/cncr.35521
Vancouver
Park KU, Lipsitz S, Dominici LS, Lynce F, Minami CA, Nakhlis F, et al. Generative artificial intelligence as a source of breast cancer information for patients: Proceed with caution. Cancer. 2025. doi:10.1002/cncr.35521
BibTeX
@article{Park2025,
author = {Park, Ko Un and Lipsitz, Stuart and Dominici, Laura S and Lynce, Filipa and Minami, Christina A and Nakhlis, Faina and Waks, Adrienne G and Warren, Laura E and Eidman, Nadine and Frazier, Jeannie and Hernandez, Lourdes and Leslie, Carla and Rafte, Susan and Stroud, Delia and Weissman, Joel S and King, Tari A and Mittendorf, Elizabeth A},
title = {Generative artificial intelligence as a source of breast cancer information for patients: Proceed with caution},
journal = {Cancer},
year = {2025},
doi = {10.1002/cncr.35521}
}
EUVIMED