EUVIMEDOpdag EUVIMED
EvaluationsstudieLeitlinieOpen access

Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study

Hsing-Yu Hsu, Lu-Wen Chen, Wan-Tseng Hsu, Yow-Wen Hsieh, Shih-Sheng Chang

Journal of Medical Internet Research · 2025-09-05 · 27 · e73486-e73486

DOI: 10.2196/73486 ↗

At a glance

Evaluationsstudie in Journal of Medical Internet Research, published online on 2025-09-05. Written by Hsing-Yu Hsu and 4 co-authors. Institution: Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan. Major MEDLINE topics: Practice Guidelines as Topic, Pharmacogenetics. Cited 12 times so far (Crossref); 27 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The effective implementation of personalized pharmacogenomics (PGx) requires the integration of released clinical guidelines into decision support systems to facilitate clinical applications. Large language models (LLMs) can be valuable tools for automating information extraction and updates.

Objective

This study aimed to assess the effectiveness of repeated cross-comparisons and an agreement-threshold strategy in 2 advanced LLMs as supportive tools for updating information.

Methods

The study evaluated the performance of 2 LLMs, GPT-4o and Gemini-1.5-Pro, in extracting PGx clinical guidelines and comparing their outputs with expert-annotated evaluations. The 2 LLMs classified 385 PGx clinical guidelines, with each recommendation tested 20 times per model. Accuracy was assessed by comparing the results with manually labeled data. Two prospectively defined strategies were used to identify inconsistent predictions. The first involved repeated cross-comparison, flagging discrepancies between the most frequent classifications from each model. The second used a consistency threshold strategy, which designated predictions appearing in less than 60% of the 40 combined outputs as unstable. Cases flagged by either strategy were subjected to manual review. This study also estimated the overall cost of model use and was conducted between October 1 and November 30, 2024.

Results

GPT-4o and Gemini-1.5-Pro yielded reproducibility rates of 97.8% (7534/7700) and 98.9% (7612/7700), respectively, based on the most frequent classification for each query. Compared with expert labels, GPT-4o achieved 93.5% accuracy (Cohen κ=0.90; P

Subjects

MeSH terms (MEDLINE)

Practice Guidelines as TopicPharmacogeneticsHumansDecision Support Systems, ClinicalLarge Language Models

Author keywords

Clinical Decision Support SystemGuideline ClassificationPharmacogenomicslarge language modelsreliabilityreproducibility

Authors and institutions

  • Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan.
  • Department of Pharmacy, China Medical University Hospital, Taichung, Taiwan.
  • Artificial Intelligence Center, China Medical University Hospital, 2, Yude Road, Taichung, 404327, Taiwan, 886 4-22052121.
  • School of Pharmacy, College of Pharmacy, China Medical University, Taichung, Taiwan.
  • Division of Cardiovascular Medicine, Department of Medicine, China Medical University Hospital, Taichung, Taiwan.
  • School of Medicine, China Medical University, Taichung, Taiwan.

Cite

APA

Hsu, H. Y., Chen, L. W., Hsu, W. T., Hsieh, Y. W., & Chang, S. S. (2025). Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research, 27, e73486-e73486. https://doi.org/10.2196/73486

Vancouver

Hsu HY, Chen LW, Hsu WT, Hsieh YW, Chang SS. Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research. 2025;27:e73486-e73486. doi:10.2196/73486

BibTeX
@article{Hsu2025,
  author = {Hsu, Hsing-Yu and Chen, Lu-Wen and Hsu, Wan-Tseng and Hsieh, Yow-Wen and Chang, Shih-Sheng},
  title = {Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e73486--e73486},
  doi = {10.2196/73486}
}
Start for free →