EUVIMEDEUVIMED entdecken
Open Access

Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis

Alyssa Aguirre iD, Robin Hilsabeck iD, Tawny Smith iD, Bo Xie iD, Daqing He iD, Zhendong Wang iD, Ning Zou iD

JMIR Aging · 06.05.2024 · 7 · e53019-e53019

DOI: 10.2196/53019 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Aging, online erschienen am 06.05.2024. Verfasst von Alyssa Aguirre und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Neurology, The University of Texas at Austin, Austin, TX, United States. Hauptthemen laut MEDLINE: Caregivers, Dementia. Die Arbeit wurde bisher 32-mal zitiert (Crossref) und nennt 12 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIA NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Artificial intelligence (AI) such as ChatGPT by OpenAI holds great promise to improve the quality of life of patients with dementia and their caregivers by providing high-quality responses to their questions about typical dementia behaviors. So far, however, evidence on the quality of such ChatGPT responses is limited. A few recent publications have investigated the quality of ChatGPT responses in other health conditions. Our study is the first to assess ChatGPT using real-world questions asked by dementia caregivers themselves.

Objectives

This pilot study examines the potential of ChatGPT-3.5 to provide high-quality information that may enhance dementia care and patient-caregiver education.

Methods

Our interprofessional team used a formal rating scale (scoring range: 0-5; the higher the score, the better the quality) to evaluate ChatGPT responses to real-world questions posed by dementia caregivers. We selected 60 posts by dementia caregivers from Reddit, a popular social media platform. These posts were verified by 3 interdisciplinary dementia clinicians as representing dementia caregivers’ desire for information in the areas of memory loss and confusion, aggression, and driving. Word count for posts in the memory loss and confusion category ranged from 71 to 531 (mean 218; median 188), aggression posts ranged from 58 to 602 words (mean 254; median 200), and driving posts ranged from 93 to 550 words (mean 272; median 276).

Results

ChatGPT’s response quality scores ranged from 3 to 5. Of the 60 responses, 26 (43%) received 5 points, 21 (35%) received 4 points, and 13 (22%) received 3 points, suggesting high quality. ChatGPT obtained consistently high scores in synthesizing information to provide follow-up recommendations (n=58, 96%), with the lowest scores in the area of comprehensiveness (n=38, 63%).

Conclusions

ChatGPT provided high-quality responses to complex questions posted by dementia caregivers, but it did have limitations. ChatGPT was unable to anticipate future problems that a human professional might recognize and address in a clinical encounter. At other times, ChatGPT recommended a strategy that the caregiver had already explicitly tried. This pilot study indicates the potential of AI to provide high-quality information to enhance dementia care and patient-caregiver education in tandem with information provided by licensed health care professionals. Evaluating the quality of responses is necessary to ensure that caregivers can make informed decisions. ChatGPT has the potential to transform health care practice by shaping how caregivers receive health information.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

CaregiversDementiaHumansPilot ProjectsQualitative ResearchMaleQuality of LifeFemaleArtificial IntelligenceAgedSocial MediaSurveys and QuestionnairesMiddle Aged

Stichwörter der Autor:innen

Alzheimer diseaseAlzheimer’s diseaseChatGPTcaregiverdementiainformation technologyneurologysocial media

Autor:innen und Institutionen

  • Department of Neurology, The University of Texas at Austin, Austin, TX, United States.
  • Steve Hicks School of Social Work, The University of Texas at Austin, Austin, TX, United States.
  • Glenn Biggs Institute for Alzheimer's & Neurodegenerative Diseases, Department of Neurology, University of Texas Health Science Center at San Antonio, San Antonio, TX, United States.
  • Department of Psychiatry and Behavioral Sciences, The University of Texas at Austin, Austin, TX, United States.
  • School of Information, The University of Texas at Austin, Austin, TX, United States.
  • School of Nursing, The University of Texas at Austin, Austin, TX, United States.

ORCID-Profile: Alyssa Aguirre, Robin Hilsabeck, Tawny Smith, Bo Xie, Daqing He, Zhendong Wang, Ning Zou

Förderung

  • NIA NIH HHS

Zitieren

APA

Aguirre, A., Hilsabeck, R., Smith, T., Xie, B., He, D., Wang, Z., & Zou, N. (2024). Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis. JMIR Aging, 7, e53019-e53019. https://doi.org/10.2196/53019

Vancouver

Aguirre A, Hilsabeck R, Smith T, Xie B, He D, Wang Z, et al. Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis. JMIR Aging. 2024;7:e53019-e53019. doi:10.2196/53019

BibTeX
@article{Aguirre2024,
  author = {Aguirre, Alyssa and Hilsabeck, Robin and Smith, Tawny and Xie, Bo and He, Daqing and Wang, Zhendong and Zou, Ning},
  title = {Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis},
  journal = {JMIR Aging},
  year = {2024},
  volume = {7},
  pages = {e53019--e53019},
  doi = {10.2196/53019}
}
Kostenlos starten →