Accuracy and readability of LLM-generated responses for gout management: a benchmark study based on ACR guidelines
Clinical Rheumatology · 2026
DOI: 10.1007/s10067-026-08073-3 ↗Kurz gefasst
Leitlinie in der Zeitschrift Clinical Rheumatology, 2026. Verfasst von Zhuhai Shao und 8 weiteren Autor:innen. Beteiligte Einrichtung: Department of Endocrine and Metabolic Diseases, the Affiliated Hospital of Qingdao University, Qingdao, China. Hauptthemen laut MEDLINE: Gout, Large Language Models, Patient Education as Topic, Practice Guidelines as Topic. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 30 Literaturangaben. Gefördert von: Natural Science Foundation of Shandong Province.
Abstract (Originalsprache)
BACKGROUND: Gout and hyperuricemia, linked to purine metabolism abnormalities or impaired uric acid excretion, are rising with lifestyle changes. Effective self-management and health literacy are crucial for gout management. While large language models (LLMs) show promise in enhancing health management, their potential on gout patient education remains underexplored. This study aimed to evaluate the accuracy and readability of responses generated by three LLMs, including DeepSeek-V3, DeepSeek-R1, and GPT-5, to questions based on the gout and hyperuricemia guidelines published by the American College of Rheumatology (ACR). METHODS: Based on the ACR gout guidelines, a set of 42 questions was curated and submitted to three LLMs. Their responses were independently rated on a 5-point Likert scale by three expert gout and hyperuricemia specialists against the guidelines. Accuracy was rated as either an average score of ≥ 4 (low threshold) or 5 (high threshold). The readability of the responses was assessed by Microsoft Word, which provided metrics including the word count, character count, Flesch Reading Ease (FRE) score, Flesch-Kincaid Grade Level (FKGL), and Automated Readability Index (ARI) were calculated. RESULTS: Our findings reveal that response accuracy was significantly higher for GPT-5 compared to DeepSeek-V3 (P 0.05). In terms of readability, GPT-5 produced the most complex responses (FKGL: 12.89 ± 2.22, ARI: 14.87 ± 2.40), while DeepSeek-R1 generated the longest outputs. CONCLUSION: LLMs show potential in generating responses that are consistent with clinical guidelines for gout management. The deployment of LLMs in gout patient education and clinical decision support necessitates the simultaneous optimization of both accuracy and readability. Key Points • This study is the first to systematically evaluate the agreement with ACR gout guidelines and readability of three state-of-the-art LLMs (DeepSeek-V3, DeepSeek-R1, GPT-5) in gout management, filling the gap of LLM benchmarking for gout patient education. • Response accuracy was significantly higher for GPT-5 compared to DeepSeek-V3, with no significant difference between GPT-5 and DeepSeek-R1. However, GPT-5 produced texts with the lowest readability. • The study innovatively combined expert-rated accuracy (5-point Likert scale by gout and hyperuricemia specialists) and objective readability metrics (FRE, FKGL, ARI), providing a comprehensive framework for assessing LLM utility in chronic disease self-management. • Findings confirm LLMs' potential for gout patient education but emphasize the need for simultaneous optimization of medical accuracy and health literacy, guiding future LLM refinement for clinical application.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Endocrine and Metabolic Diseases, the Affiliated Hospital of Qingdao University, Qingdao, China.
- Department of Endocrinology and Metabolism, West China Hospital, Sichuan University, Sichuan, China.
- Qingdao West Coast New Area District Hospital, Qingdao, China.
- Department of Endocrine and Metabolic Diseases, the Affiliated Hospital of Qingdao University, Qingdao, China. qdunfmk@163.com.
- Department of Endocrine and Metabolic Diseases, the Affiliated Hospital of Qingdao University, Qingdao, China. qdumzp@hotmail.com.
Förderung
- Natural Science Foundation of Shandong Province
Zitieren
APA
Shao, Z., Zhang, Y., Cheng, B., Zhang, F., Zhu, X., Wang, Z., Wang, Y., Xu, L., & Mu, Z. (2026). Accuracy and readability of LLM-generated responses for gout management: a benchmark study based on ACR guidelines. Clinical Rheumatology. https://doi.org/10.1007/s10067-026-08073-3
Vancouver
Shao Z, Zhang Y, Cheng B, Zhang F, Zhu X, Wang Z, et al. Accuracy and readability of LLM-generated responses for gout management: a benchmark study based on ACR guidelines. Clinical Rheumatology. 2026. doi:10.1007/s10067-026-08073-3
BibTeX
@article{Shao2026,
author = {Shao, Zhuhai and Zhang, Yiwen and Cheng, Bingfei and Zhang, Fang and Zhu, Xuhua and Wang, Zhongchao and Wang, Yangang and Xu, Lili and Mu, Zepeng},
title = {Accuracy and readability of LLM-generated responses for gout management: a benchmark study based on ACR guidelines},
journal = {Clinical Rheumatology},
year = {2026},
doi = {10.1007/s10067-026-08073-3}
}
EUVIMED