EUVIMEDEUVIMED entdecken
EvaluationsstudieLeitlinieOpen Access

Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study

Hsing-Yu Hsu, Lu-Wen Chen, Wan-Tseng Hsu, Yow-Wen Hsieh, Shih-Sheng Chang

Journal of Medical Internet Research · 05.09.2025 · 27 · e73486-e73486

DOI: 10.2196/73486 ↗

Kurz gefasst

Evaluationsstudie in der Zeitschrift Journal of Medical Internet Research, online erschienen am 05.09.2025. Verfasst von Hsing-Yu Hsu und 4 weiteren Autor:innen. Beteiligte Einrichtung: Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan. Hauptthemen laut MEDLINE: Practice Guidelines as Topic, Pharmacogenetics. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 27 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The effective implementation of personalized pharmacogenomics (PGx) requires the integration of released clinical guidelines into decision support systems to facilitate clinical applications. Large language models (LLMs) can be valuable tools for automating information extraction and updates.

Objective

This study aimed to assess the effectiveness of repeated cross-comparisons and an agreement-threshold strategy in 2 advanced LLMs as supportive tools for updating information.

Methods

The study evaluated the performance of 2 LLMs, GPT-4o and Gemini-1.5-Pro, in extracting PGx clinical guidelines and comparing their outputs with expert-annotated evaluations. The 2 LLMs classified 385 PGx clinical guidelines, with each recommendation tested 20 times per model. Accuracy was assessed by comparing the results with manually labeled data. Two prospectively defined strategies were used to identify inconsistent predictions. The first involved repeated cross-comparison, flagging discrepancies between the most frequent classifications from each model. The second used a consistency threshold strategy, which designated predictions appearing in less than 60% of the 40 combined outputs as unstable. Cases flagged by either strategy were subjected to manual review. This study also estimated the overall cost of model use and was conducted between October 1 and November 30, 2024.

Results

GPT-4o and Gemini-1.5-Pro yielded reproducibility rates of 97.8% (7534/7700) and 98.9% (7612/7700), respectively, based on the most frequent classification for each query. Compared with expert labels, GPT-4o achieved 93.5% accuracy (Cohen κ=0.90; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Practice Guidelines as TopicPharmacogeneticsHumansDecision Support Systems, ClinicalLarge Language Models

Stichwörter der Autor:innen

Clinical Decision Support SystemGuideline ClassificationPharmacogenomicslarge language modelsreliabilityreproducibility

Autor:innen und Institutionen

  • Graduate Institute of Clinical Pharmacy, College of Medicine, National Taiwan University, Taipei, Taiwan.
  • Department of Pharmacy, China Medical University Hospital, Taichung, Taiwan.
  • Artificial Intelligence Center, China Medical University Hospital, 2, Yude Road, Taichung, 404327, Taiwan, 886 4-22052121.
  • School of Pharmacy, College of Pharmacy, China Medical University, Taichung, Taiwan.
  • Division of Cardiovascular Medicine, Department of Medicine, China Medical University Hospital, Taichung, Taiwan.
  • School of Medicine, China Medical University, Taichung, Taiwan.

Zitieren

APA

Hsu, H. Y., Chen, L. W., Hsu, W. T., Hsieh, Y. W., & Chang, S. S. (2025). Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research, 27, e73486-e73486. https://doi.org/10.2196/73486

Vancouver

Hsu HY, Chen LW, Hsu WT, Hsieh YW, Chang SS. Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study. Journal of Medical Internet Research. 2025;27:e73486-e73486. doi:10.2196/73486

BibTeX
@article{Hsu2025,
  author = {Hsu, Hsing-Yu and Chen, Lu-Wen and Hsu, Wan-Tseng and Hsieh, Yow-Wen and Chang, Shih-Sheng},
  title = {Extracting Clinical Guideline Information Using Two Large Language Models: Evaluation Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e73486--e73486},
  doi = {10.2196/73486}
}
Kostenlos starten →