EUVIMEDOpdag EUVIMED
Open access

Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis

Alyssa Aguirre iD, Robin Hilsabeck iD, Tawny Smith iD, Bo Xie iD, Daqing He iD, Zhendong Wang iD, Ning Zou iD

JMIR Aging · 2024-05-06 · 7 · e53019-e53019

DOI: 10.2196/53019 ↗

At a glance

Journal article in JMIR Aging, published online on 2024-05-06. Written by Alyssa Aguirre and 6 co-authors. Institution: Department of Neurology, The University of Texas at Austin, Austin, TX, United States. Major MEDLINE topics: Caregivers, Dementia. Cited 32 times so far (Crossref); 12 references. The full text is freely available in PubMed Central. Funded by: NIA NIH HHS.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Artificial intelligence (AI) such as ChatGPT by OpenAI holds great promise to improve the quality of life of patients with dementia and their caregivers by providing high-quality responses to their questions about typical dementia behaviors. So far, however, evidence on the quality of such ChatGPT responses is limited. A few recent publications have investigated the quality of ChatGPT responses in other health conditions. Our study is the first to assess ChatGPT using real-world questions asked by dementia caregivers themselves.

Objectives

This pilot study examines the potential of ChatGPT-3.5 to provide high-quality information that may enhance dementia care and patient-caregiver education.

Methods

Our interprofessional team used a formal rating scale (scoring range: 0-5; the higher the score, the better the quality) to evaluate ChatGPT responses to real-world questions posed by dementia caregivers. We selected 60 posts by dementia caregivers from Reddit, a popular social media platform. These posts were verified by 3 interdisciplinary dementia clinicians as representing dementia caregivers’ desire for information in the areas of memory loss and confusion, aggression, and driving. Word count for posts in the memory loss and confusion category ranged from 71 to 531 (mean 218; median 188), aggression posts ranged from 58 to 602 words (mean 254; median 200), and driving posts ranged from 93 to 550 words (mean 272; median 276).

Results

ChatGPT’s response quality scores ranged from 3 to 5. Of the 60 responses, 26 (43%) received 5 points, 21 (35%) received 4 points, and 13 (22%) received 3 points, suggesting high quality. ChatGPT obtained consistently high scores in synthesizing information to provide follow-up recommendations (n=58, 96%), with the lowest scores in the area of comprehensiveness (n=38, 63%).

Conclusions

ChatGPT provided high-quality responses to complex questions posted by dementia caregivers, but it did have limitations. ChatGPT was unable to anticipate future problems that a human professional might recognize and address in a clinical encounter. At other times, ChatGPT recommended a strategy that the caregiver had already explicitly tried. This pilot study indicates the potential of AI to provide high-quality information to enhance dementia care and patient-caregiver education in tandem with information provided by licensed health care professionals. Evaluating the quality of responses is necessary to ensure that caregivers can make informed decisions. ChatGPT has the potential to transform health care practice by shaping how caregivers receive health information.

Subjects

MeSH terms (MEDLINE)

CaregiversDementiaHumansPilot ProjectsQualitative ResearchMaleQuality of LifeFemaleArtificial IntelligenceAgedSocial MediaSurveys and QuestionnairesMiddle Aged

Author keywords

Alzheimer diseaseAlzheimer’s diseaseChatGPTcaregiverdementiainformation technologyneurologysocial media

Authors and institutions

  • Department of Neurology, The University of Texas at Austin, Austin, TX, United States.
  • Steve Hicks School of Social Work, The University of Texas at Austin, Austin, TX, United States.
  • Glenn Biggs Institute for Alzheimer's & Neurodegenerative Diseases, Department of Neurology, University of Texas Health Science Center at San Antonio, San Antonio, TX, United States.
  • Department of Psychiatry and Behavioral Sciences, The University of Texas at Austin, Austin, TX, United States.
  • School of Information, The University of Texas at Austin, Austin, TX, United States.
  • School of Nursing, The University of Texas at Austin, Austin, TX, United States.

ORCID profiles: Alyssa Aguirre, Robin Hilsabeck, Tawny Smith, Bo Xie, Daqing He, Zhendong Wang, Ning Zou

Funding

  • NIA NIH HHS

Cite

APA

Aguirre, A., Hilsabeck, R., Smith, T., Xie, B., He, D., Wang, Z., & Zou, N. (2024). Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis. JMIR Aging, 7, e53019-e53019. https://doi.org/10.2196/53019

Vancouver

Aguirre A, Hilsabeck R, Smith T, Xie B, He D, Wang Z, et al. Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis. JMIR Aging. 2024;7:e53019-e53019. doi:10.2196/53019

BibTeX
@article{Aguirre2024,
  author = {Aguirre, Alyssa and Hilsabeck, Robin and Smith, Tawny and Xie, Bo and He, Daqing and Wang, Zhendong and Zou, Ning},
  title = {Assessing the Quality of ChatGPT Responses to Dementia Caregivers’ Questions: Qualitative Analysis},
  journal = {JMIR Aging},
  year = {2024},
  volume = {7},
  pages = {e53019--e53019},
  doi = {10.2196/53019}
}
Start for free →