EUVIMEDOdkrijte EUVIMED
Randomisierte kontrollierte StudieMixed-Methods-StudieOpen access

Effects of a Personalized Retrieval-Augmented Generation Chatbot on Information Needs in Patients With Chronic Kidney Disease: Mixed Methods Randomized Controlled Trial

Hee Jeong Hwang, Junghwan Kim, Dongseok Heo, Jeong Yun You, Jeonghwan Lee, Soie Kwon, Kyung Don Yoo, Jung Han Yoon Park, Bongwon Suh, Jung Pyo Lee, Ki Won Lee

Interactive Journal of Medical Research · 2026-11-13 · 15 · e96501-e96501

DOI: 10.2196/96501 ↗

At a glance

Randomisierte kontrollierte Studie in Interactive Journal of Medical Research, published online on 2026-11-13. Written by Hee Jeong Hwang and 10 co-authors. Institution: Department of Agricultural Biotechnology, College of Agriculture and Life Sciences, Seoul National University, 1, Gwanak-ro, Gwanak-gu, Seoul, Seoul,…. 43 references. The full text is freely available in PubMed Central.

PovzetekSubjectsBibliografski podatkiCiteRelated studies

Abstract (original language)

Background

Patients receiving dialysis for chronic kidney disease encounter complex self-management challenges and ongoing information needs that traditional health care often fails to meet. Retrieval-augmented generation (RAG) enhances the factual accuracy of large language models (LLMs), but whether personalized RAG is more effective than general RAG remains unclear.

Objective

This study aimed to assess whether personalized constraint retrieval-augmented generation (C-RAG) is more effective than open-domain general retrieval-augmented generation (O-RAG) and standard care in reducing information needs among patients receiving dialysis and to identify predictors of chatbot engagement.

Methods

A sequential 2-phase mixed methods study was conducted. Phase 1 evaluated 4 LLMs (Claude 3.5 Sonnet [Anthropic], Gemini 1.5 Flash [Google], CLOVA X [NAVER], and ChatGPT-4o [OpenAI]) using 60 patient-generated questions, assessed by 15 clinical experts. Phase 2 was a 3-arm parallel pilot randomized controlled trial (RCT; October-December 2025) at 3 tertiary hospitals in South Korea. Based on phase 1 findings, 2 chatbot variants were developed using GPT-4o—C-RAG, which incorporates patient-specific clinical parameters, and O-RAG, which uses the same knowledge base without personalization. Participants were randomized 1:1:1 to C-RAG, O-RAG, or standard care. The primary outcome, change in medication, dietary, and diagnostic information needs (5-point Likert scales) from baseline to 4 weeks, was analyzed using ANCOVA. Semistructured interviews with 9 participants explored underlying mechanisms.

Results

ChatGPT-4o achieved the highest scores in clinical accuracy (mean 3.60, SD 0.95), safety (mean 3.67, SD 1.00), and readability (Flesch-Kincaid Grade Level 9.8, compared with 13.2‐14.5 for other models). In phase 2, 42/45 (93%) enrolled participants completed the study (14 per group). ANCOVA revealed significant group differences in medication ( F 2,38 =4.574, P =.02) and dietary ( F 2,38 =4.232, P =.02) information needs. In Bonferroni-corrected comparisons, C-RAG had lower needs than O-RAG in both domains (observed mean 2.93, SD 1.21 vs mean 3.86, SD 0.95; Cohen d =−0.86, 95% CI −1.63 to −0.08; P =.02 for both). Against control, C-RAG showed medium-to-large but nonsignificant effects (medication Cohen d =−0.64, 95% CI −1.39 to 0.12; dietary Cohen d =−0.54, 95% CI −1.30 to 0.21). Secondary outcomes did not differ between groups. Dietary management emerged as the most frequent query topic, accounting for 29.3% (205/691) of total queries. Participants on dialysis for 1‐3 years showed higher engagement (9/10, 90% engagement for 1‐3 y vs 5/18, 27.8% for ≥4 y; Fisher exact P =.006). Qualitative analysis revealed 5 themes, including dietary management as the primary need and dialysis duration as a moderator of utility.

Conclusions

In this pilot trial, personalized RAG showed lower information needs than general RAG, with a large effect size (Cohen d =−0.86). The medium-to-large effects against standard care (Cohen d =−0.54 to −0.64) support a fully powered confirmatory trial (estimated 50‐55 participants per group) to evaluate long-term clinical outcomes.

Subjects

Author keywords

chatbotschronic kidney diseasedietary managementdigital healthinformation needslarge language modelsrandomized controlled trial​​​​​​​​​​​​​​​​renal dialysisretrieval-augmented generationself-management

Authors and institutions

  • Department of Agricultural Biotechnology, College of Agriculture and Life Sciences, Seoul National University, 1, Gwanak-ro, Gwanak-gu, Seoul, Seoul, 08826, Republic of Korea, 82 10-9986-7464.
  • Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, Seoul, Republic of Korea.
  • Department of Internal Medicine, College of Medicine, Seoul National University, Seoul, Seoul, Republic of Korea.
  • Department of Internal Medicine, SNUH SMG-SNU Boramae Medical Center, Seoul, Seoul, Republic of Korea.
  • Department of Internal Medicine, Chung-Ang University Hospital, Seoul, Seoul, Republic of Korea.
  • Department of Internal Medicine, Chung-Ang University, Seoul, Seoul, Republic of Korea.

Cite

APA

Hwang, H. J., Kim, J., Heo, D., You, J. Y., Lee, J., Kwon, S., Yoo, K. D., Yoon Park, J. H., Suh, B., Lee, J. P., & Lee, K. W. (2026). Effects of a Personalized Retrieval-Augmented Generation Chatbot on Information Needs in Patients With Chronic Kidney Disease: Mixed Methods Randomized Controlled Trial. Interactive Journal of Medical Research, 15, e96501-e96501. https://doi.org/10.2196/96501

Vancouver

Hwang HJ, Kim J, Heo D, You JY, Lee J, Kwon S, et al. Effects of a Personalized Retrieval-Augmented Generation Chatbot on Information Needs in Patients With Chronic Kidney Disease: Mixed Methods Randomized Controlled Trial. Interactive Journal of Medical Research. 2026;15:e96501-e96501. doi:10.2196/96501

BibTeX
@article{Hwang2026,
  author = {Hwang, Hee Jeong and Kim, Junghwan and Heo, Dongseok and You, Jeong Yun and Lee, Jeonghwan and Kwon, Soie and Yoo, Kyung Don and Yoon Park, Jung Han and Suh, Bongwon and Lee, Jung Pyo and Lee, Ki Won},
  title = {Effects of a Personalized Retrieval-Augmented Generation Chatbot on Information Needs in Patients With Chronic Kidney Disease: Mixed Methods Randomized Controlled Trial},
  journal = {Interactive Journal of Medical Research},
  year = {2026},
  volume = {15},
  pages = {e96501--e96501},
  doi = {10.2196/96501}
}
Start for free →