Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis
Otolaryngology–Head and Neck Surgery · 2024-08 · 171 (2) · 571-577
DOI: 10.1002/ohn.832 ↗At a glance
Journal article in Otolaryngology–Head and Neck Surgery, published online on 2024-05-26. Written by Fan Ye and 5 co-authors. Institution: Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China. Major MEDLINE topics: Sinusitis, Rhinitis, Allergic. Cited 20 times so far (Crossref); 17 references.
Abstract (original language)
Objective
This study aims to evaluate the accuracy of ChatGPT in answering allergic rhinitis (AR) and chronic rhinosinusitis (CRS) related questions. Study Design This is a cross‐sectional study.
Setting
Each question was inputted as a separate, independent prompt.
Methods
Responses to AR (n = 189) and CRS (n = 242) related questions, generated by GPT‐3.5 and GPT‐4, were independently graded for accuracy by 2 senior rhinology professors, with disagreements adjudicated by a third reviewer.
Results
Overall, ChatGPT demonstrated a satisfactory performance, accurately answering over 80% of questions across all categories. Specifically, GPT‐4.0's accuracy in responding to AR‐related questions significantly exceeded that of GPT‐3.5, but distinction not evident in CRS‐related questions. Patient‐originated questions had a significantly higher accuracy compared to doctor‐originated questions when utilizing GPT‐4.0 to respond to AR‐related questions. This discrepancy was not observed with GPT‐3.5 or in the context of CRS‐related questions. Across different types of content, ChatGPT excelled in covering basic knowledge, prevention, and emotion for AR and CRS. However, it experienced challenges when addressing questions about recent advancements, a trend consistent across both GPT‐3.5 and GPT‐4.0 iterations. Importantly, the accuracy of responses remained unaffected when questions were posed in Chinese.
Conclusion
Our findings suggest ChatGPT's capability to convey accurate information for AR and CRS patients, and offer insights into its performance across various domains, guiding its utilization and improvement.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Otolaryngology-Head and Neck Surgery, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China.
- Department of Allergy, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China.
- Naso-Orbital-Maxilla and Skull Base Center, The Third Affiliated Hospital of Sun Yat-Sen University, Guangzhou, China.
- Key Laboratory of Airway Inflammatory Disease Research and Innovative Technology Translation, Guangzhou, China.
Cite
APA
Ye, F., Zhang, H., Luo, X., Wu, T., Yang, Q., & Shi, Z. (2024). Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis. Otolaryngology–Head and Neck Surgery, 171(2), 571-577. https://doi.org/10.1002/ohn.832
Vancouver
Ye F, Zhang H, Luo X, Wu T, Yang Q, Shi Z. Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis. Otolaryngology–Head and Neck Surgery. 2024;171(2):571-577. doi:10.1002/ohn.832
BibTeX
@article{Ye2024,
author = {Ye, Fan and Zhang, He and Luo, Xin and Wu, Tong and Yang, Qintai and Shi, Zhaohui},
title = {Evaluating ChatGPT's Performance in Answering Questions About Allergic Rhinitis and Chronic Rhinosinusitis},
journal = {Otolaryngology–Head and Neck Surgery},
year = {2024},
volume = {171},
number = {2},
pages = {571--577},
doi = {10.1002/ohn.832}
}
EUVIMED