Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study
Journal of Medical Internet Research · 2025-09-05 · 27 · e73486-e73486
DOI: 10.2196/73486 ↗At a glance
Evaluationsstudie in Journal of Medical Internet Research, published online on 2025-09-05. Written by Hsing-Yu Hsu and 4 co-authors. Institution: Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan. Major MEDLINE topics: Practice Guidelines as Topic, Pharmacogenetics. Cited 12 times so far (Crossref); 27 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
The effective implementation of personalized pharmacogenomics (PGx) requires the integration of released clinical guidelines into decision support systems to facilitate clinical applications. Large language models (LLMs) can be valuable tools for automating information extraction and updates.
Objective
This study aimed to assess the effectiveness of repeated cross-comparisons and an agreement-threshold strategy in 2 advanced LLMs as supportive tools for updating information.
Methods
The study evaluated the performance of 2 LLMs, GPT-4o and Gemini-1.5-Pro, in extracting PGx clinical guidelines and comparing their outputs with expert-annotated evaluations. The 2 LLMs classified 385 PGx clinical guidelines, with each recommendation tested 20 times per model. Accuracy was assessed by comparing the results with manually labeled data. Two prospectively defined strategies were used to identify inconsistent predictions. The first involved repeated cross-comparison, flagging discrepancies between the most frequent classifications from each model. The second used a consistency threshold strategy, which designated predictions appearing in less than 60% of the 40 combined outputs as unstable. Cases flagged by either strategy were subjected to manual review. This study also estimated the overall cost of model use and was conducted between October 1 and November 30, 2024.
Results
GPT-4o and Gemini-1.5-Pro yielded reproducibility rates of 97.8% (7534/7700) and 98.9% (7612/7700), respectively, based on the most frequent classification for each query. Compared with expert labels, GPT-4o achieved 93.5% accuracy (Cohen κ=0.90; P
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan.
- Department of Pharmacy, China Medical University Hospital, Taichung, Taiwan.
- Artificial Intelligence Center, China Medical University Hospital, 2, Yude Road, Taichung, 404327, Taiwan, 886 4-22052121.
- School of Pharmacy, College of Pharmacy, China Medical University, Taichung, Taiwan.
- Division of Cardiovascular Medicine, Department of Medicine, China Medical University Hospital, Taichung, Taiwan.
- School of Medicine, China Medical University, Taichung, Taiwan.
Cite
APA
Hsu, H. Y., Chen, L. W., Hsu, W. T., Hsieh, Y. W., & Chang, S. S. (2025). Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research, 27, e73486-e73486. https://doi.org/10.2196/73486
Vancouver
Hsu HY, Chen LW, Hsu WT, Hsieh YW, Chang SS. Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research. 2025;27:e73486-e73486. doi:10.2196/73486
BibTeX
@article{Hsu2025,
author = {Hsu, Hsing-Yu and Chen, Lu-Wen and Hsu, Wan-Tseng and Hsieh, Yow-Wen and Chang, Shih-Sheng},
title = {Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e73486--e73486},
doi = {10.2196/73486}
}
EUVIMED