EUVIMEDSkopri EUVIMED
Beobachtungsstudie

Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study

Qiuhong Wei iD, Li Xiao iD, Xin Zhang iD, Hong Wei iD, Ruijue Wang iD, Suqiao Ou iD, Chao Zhang iD, Huarong Huo iD, Yu Peng iD, Lishen Liu iD, Ping Song iD, Ziyu Hua iD and 1 more

Journal of Medical Internet Research · 2026-11-04 · 28 · e105346

DOI: 10.2196/105346 ↗

At a glance

Beobachtungsstudie in Journal of Medical Internet Research, published online on 2026-11-04. Written by Qiuhong Wei and 12 co-authors. Institution: Big Data Center for Children's Medical Care, Children's Hospital of Chongqing Medical University, Chongqing, China. Major MEDLINE topics: Large Language Models, Pediatrics. 38 references.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Pediatric care resources are limited and unevenly distributed in many health systems. Caregivers increasingly turn to large language models (LLMs) for children’s health information, but real-world evidence on LLM use and response quality in general pediatric care is limited.

Objective

This study aimed to characterize real-world use, consultation needs, and response quality of Xiaoyi, an LLM-based pediatric health assistant, and to evaluate the impact of expert-guided optimization on response quality.

Methods

This study collected all consultation sessions submitted between April 7 and December 31, 2025, to Xiaoyi, a pediatric health assistant co-developed by Children’s Hospital of Chongqing Medical University and Zuoyi Tech. Temporal patterns of consultation activity were characterized; all sessions were annotated for pediatric subspecialty, reason for consultation, and presenting symptoms through LLM-assisted labeling with pediatrician validation. A multidisciplinary panel of 40 pediatricians evaluated Xiaoyi’s response quality over 2 rounds using predefined criteria established by expert consensus: an initial evaluation informed system optimization (prompt revision, knowledge-base and tool expansion, and validation rules) and a follow-up evaluation assessed its impact. In each round, 1000 sessions randomly sampled from the corresponding pre- or postoptimization period (2000 in total) were rated at the session level (overall quality and clinical appropriateness) and the turn level (6 predefined deficiency categories). Categorical ratings were compared using the chi-square test, and turn-level deficiency rates were compared using generalized estimating equations to account for within-session correlation.

Results

Caregivers of 8266 children (median age 4.2, IQR 1.3-8 years; 4705/8266, 56.9% male) from 31 of 34 provincial-level divisions in China initiated 16,074 sessions, of which 7046 (43.8%) occurred outside routine clinic hours (08:00-17:30). Queries spanned 31 subspecialties, most frequently child health care (2025/16,074, 12.6%), pulmonology (1903/16,074, 11.8%), and gastroenterology (1833/16,074, 11.4%). The leading reasons for consultation were symptom consultation (7845/16,074, 48.8%), medical report interpretation (2531/16,074, 15.7%), and triage guidance (2515/16,074, 15.6%); among symptom consultation sessions, fever (2719/16,074, 34.7%) and cough (2349/16,074, 29.9%) were most common. After expert-guided optimization, the number of sessions rated good (n=1000) in overall quality (the response fully resolving the caregiver’s question with actionable information) increased from 652 (65.2%) to 743 (74.3%), and the number rated clinically appropriate increased from 614 (61.4%) to 874 (87.4%, both P

Subjects

MeSH terms (MEDLINE)

Large Language ModelsPediatricsHumansRetrospective StudiesChild

Author keywords

expert-guided optimizationhealth consultationlarge language modelpediatricsreal-world study

Authors and institutions

  • Big Data Center for Children's Medical Care, Children's Hospital of Chongqing Medical University, Chongqing, China.
  • Department of Children's Healthcare, Children's Hospital of Chongqing Medical University, Chongqing, China.
  • National Clinical Research Center for Children and Adolescents' Health and Diseases, Chongqing, China.
  • Intelligent Application of Big Data in Pediatrics Engineering Research Center of Chongqing Education Commission, Chongqing, China.
  • Ministry of Education Key Laboratory of Child Development and Disorders, Chongqing, China.
  • Department of Hematology, Children's Hospital of Chongqing Medical University, Chongqing, China.

ORCID profiles: Qiuhong Wei, Li Xiao, Xin Zhang, Hong Wei, Ruijue Wang, Suqiao Ou, Chao Zhang, Huarong Huo, Yu Peng, Lishen Liu, Ping Song, Ziyu Hua, Ximing Xu

Cite

APA

Wei, Q., Xiao, L., Zhang, X., Wei, H., Wang, R., Ou, S., Zhang, C., Huo, H., Peng, Y., Liu, L., Song, P., Hua, Z., & Xu, X. (2026). Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study. Journal of Medical Internet Research, 28, e105346. https://doi.org/10.2196/105346

Vancouver

Wei Q, Xiao L, Zhang X, Wei H, Wang R, Ou S, et al. Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study. Journal of Medical Internet Research. 2026;28:e105346. doi:10.2196/105346

BibTeX
@article{Wei2026,
  author = {Wei, Qiuhong and Xiao, Li and Zhang, Xin and Wei, Hong and Wang, Ruijue and Ou, Suqiao and Zhang, Chao and Huo, Huarong and Peng, Yu and Liu, Lishen and Song, Ping and Hua, Ziyu and Xu, Ximing},
  title = {Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study},
  journal = {Journal of Medical Internet Research},
  year = {2026},
  volume = {28},
  pages = {e105346},
  doi = {10.2196/105346}
}
Start for free →