Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study
Journal of Medical Internet Research · 05.09.2025 · 27 · e73486-e73486
DOI: 10.2196/73486 ↗Kurz gefasst
Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 05.09.2025. Verfasst von Hsing-Yu Hsu und 4 weiteren Autor:innen. Beteiligte Einrichtung: Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan. Hauptthemen laut MEDLINE: Practice Guidelines as Topic, Pharmacogenetics. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
The effective implementation of personalized pharmacogenomics (PGx) requires the integration of released clinical guidelines into decision support systems to facilitate clinical applications. Large language models (LLMs) can be valuable tools for automating information extraction and updates.
Objective
This study aimed to assess the effectiveness of repeated cross-comparisons and an agreement-threshold strategy in 2 advanced LLMs as supportive tools for updating information.
Methods
The study evaluated the performance of 2 LLMs, GPT-4o and Gemini-1.5-Pro, in extracting PGx clinical guidelines and comparing their outputs with expert-annotated evaluations. The 2 LLMs classified 385 PGx clinical guidelines, with each recommendation tested 20 times per model. Accuracy was assessed by comparing the results with manually labeled data. Two prospectively defined strategies were used to identify inconsistent predictions. The first involved repeated cross-comparison, flagging discrepancies between the most frequent classifications from each model. The second used a consistency threshold strategy, which designated predictions appearing in less than 60% of the 40 combined outputs as unstable. Cases flagged by either strategy were subjected to manual review. This study also estimated the overall cost of model use and was conducted between October 1 and November 30, 2024.
Results
GPT-4o and Gemini-1.5-Pro yielded reproducibility rates of 97.8% (7534/7700) and 98.9% (7612/7700), respectively, based on the most frequent classification for each query. Compared with expert labels, GPT-4o achieved 93.5% accuracy (Cohen κ=0.90; P
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan.
- Department of Pharmacy, China Medical University Hospital, Taichung, Taiwan.
- Artificial Intelligence Center, China Medical University Hospital, 2, Yude Road, Taichung, 404327, Taiwan, 886 4-22052121.
- School of Pharmacy, College of Pharmacy, China Medical University, Taichung, Taiwan.
- Division of Cardiovascular Medicine, Department of Medicine, China Medical University Hospital, Taichung, Taiwan.
- School of Medicine, China Medical University, Taichung, Taiwan.
Zitieren
APA
Hsu, H. Y., Chen, L. W., Hsu, W. T., Hsieh, Y. W., & Chang, S. S. (2025). Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research, 27, e73486-e73486. https://doi.org/10.2196/73486
Vancouver
Hsu HY, Chen LW, Hsu WT, Hsieh YW, Chang SS. Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research. 2025;27:e73486-e73486. doi:10.2196/73486
BibTeX
@article{Hsu2025,
author = {Hsu, Hsing-Yu and Chen, Lu-Wen and Hsu, Wan-Tseng and Hsieh, Yow-Wen and Chang, Shih-Sheng},
title = {Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study},
journal = {Journal of Medical Internet Research},
year = {2025},
volume = {27},
pages = {e73486--e73486},
doi = {10.2196/73486}
}
EUVIMED