Evaluating the comprehension and accuracy of ChatGPT's responses to diabetes-related questions in Urdu compared to English
DIGITAL HEALTH · 2024-01 · 10
DOI: 10.1177/20552076241289730 ↗Kurz gefasst
Fachartikel in der Zeitschrift DIGITAL HEALTH, online erschienen am 17.10.2024. Verfasst von Seyreen Faisal und 8 weiteren Autor:innen. Beteiligte Einrichtung: Shifa College of Medicine, Shifa Tameer-e-Millat University, Islamabad, Pakistan. Die Arbeit wurde bisher 10-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).
Themen: Diabetes mellitus, Künstliche Intelligenz in der Pflege
Abstract (Originalsprache)
Introduction
Patients with diabetes require healthcare and information that are accurate and extensive. Large language models (LLMs) like ChatGPT herald the capacity to provide such exhaustive data. To determine (a) the comprehensiveness of ChatGPT's responses in Urdu to diabetes-related questions and (b) the accuracy of ChatGPT's Urdu responses when compared to its English responses.
Methods
A cross-sectional observational study was conducted. Two reviewers experienced in internal medicine and endocrinology graded 53 Urdu and English responses on diabetes knowledge, lifestyle, and prevention. A senior reviewer resolved discrepancies. Responses were assessed for comprehension and accuracy, then compared to English.
Results
Among the Urdu responses generated, only two of 53 (3.8%) questions were graded as comprehensive, and five of 53 (9.4%) were graded as correct but inadequate. We found that 25 of 53 (47.2%) questions were graded as mixed with correct and incorrect/outdated data, the most significant proportion of responses being graded as such. When considering the comparison of response scale grading the comparative accuracy of Urdu and English responses, no Urdu response (0.0%) was considered to have more accuracy than English. Most of the Urdu responses were found to have an accuracy less than that of English, an overwhelming majority of 49 of 53 (92.5%) responses.
Conclusion
We found that although the ability to retrieve such information about diabetes is impressive, it can merely be used as an adjunct instead of a solitary source of information. Further work must be done to optimize Urdu responses in medical contexts to approximate the boundless potential it heralds.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Shifa College of Medicine, Shifa Tameer-e-Millat University, Islamabad, Pakistan.
- Department of Internal Medicine, Shifa College of Medicine, Shifa Tameer-e-Millat University, Islamabad, Pakistan.
- Islamabad Medical and Dental College, Shaheed Zulfiqar Ali Bhutto Medical University, Islamabad, Pakistan.
- Department of Endocrinology, Shifa International Hospital, Islamabad, Pakistan.
ORCID-Profile: Seyreen Faisal
Zitieren
APA
Faisal, S., Kamran, T. E., Khalid, R., Haider, Z., Siddiqui, Y., Saeed, N., Imran, S., Faisal, R., & Jabeen, M. (2024). Evaluating the comprehension and accuracy of ChatGPT's responses to diabetes-related questions in Urdu compared to English. DIGITAL HEALTH, 10. https://doi.org/10.1177/20552076241289730
Vancouver
Faisal S, Kamran TE, Khalid R, Haider Z, Siddiqui Y, Saeed N, et al. Evaluating the comprehension and accuracy of ChatGPT's responses to diabetes-related questions in Urdu compared to English. DIGITAL HEALTH. 2024;10. doi:10.1177/20552076241289730
BibTeX
@article{Faisal2024,
author = {Faisal, Seyreen and Kamran, Tafiya Erum and Khalid, Rimsha and Haider, Zaira and Siddiqui, Yusra and Saeed, Nadia and Imran, Sunaina and Faisal, Romaan and Jabeen, Misbah},
title = {Evaluating the comprehension and accuracy of ChatGPT's responses to diabetes-related questions in Urdu compared to English},
journal = {DIGITAL HEALTH},
year = {2024},
volume = {10},
doi = {10.1177/20552076241289730}
}
EUVIMED