EUVIMEDEUVIMED entdecken
VergleichsstudieBeobachtungsstudieOpen Access · CC BY-NC 4.0

Comparison of Large Language Model‐Based Systems and Prompt Engineering for Internal Medicine Clinical Pharmacy Cases

Samuel S Yang iD, Clement E Ng iD, Hyunuk Seung iD, Sean Kelly

JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY · 2026-07 · 9 (7)

DOI: 10.1002/jac5.70235 ↗

Kurz gefasst

Vergleichsstudie in der Zeitschrift JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY, online erschienen am 21.06.2026. Verfasst von Samuel S Yang und 3 weiteren Autor:innen. Beteiligte Einrichtung: University of Maryland Medical Center, Baltimore, Maryland, USA. Hauptthemen laut MEDLINE: Large Language Models, Internal Medicine, Pharmacy Service, Hospital, Clinical Pharmacy Information Systems. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 35 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Large language models (LLMs) are an application of artificial intelligence and generate responses to user inquiries that vary in accuracy and completeness. Methods of improving LLM response quality are poorly evaluated in the context of clinical pharmacy practice.

Methods

This was a single‐center, observational, prospective study conducted through internal medicine admitting services. A clinical pharmacy specialist developed 50 case questions reflective of usual practice, which were processed through LLM‐based systems in a two‐by‐two factorial design. The first variable was the selection of a general‐purpose LLM, ChatGPT 4o (GPT, OpenAI Inc., San Francisco, California, USA), or a health care provider domain‐specific LLM with retrieval‐augmented generation (RAG) features, OpenEvidence (OpenEvidence, Miami, Florida, USA). The second variable was the inclusion of a prompt engineering template with specific instructions and parameters to refine the system output. Responses were evaluated by two pharmacists and reconciled with a third pharmacist. The primary endpoint was a composite of response accuracy and completeness. Secondary outcomes included accuracy, completeness, reference validity, reproducibility, and extraneous information.

Results

Logistic regression modeling demonstrated no statistically significant interactions between the two LLM‐based systems and use of a prompt engineering template for accuracy and completeness. Predicted probabilities for meeting the primary outcome were as follows: GPT no Template 0.54, GPT with Template 0.60, OpenEvidence no Template 0.64, and OpenEvidence with Template 0.52. OpenEvidence reference validity was higher than GPT regardless of prompt engineering template use ( p < 0.001 for all comparisons).

Conclusion

Neither the use of health care provider domain‐specific LLM with RAG nor a prompt engineering template was found to improve LLM‐based system accuracy and completeness. OpenEvidence's ability to cite relevant and correct references more frequently than GPT shows promise for practice applications. There is a need for further evaluation of methods to improve LLM utility as artificial intelligence becomes further integrated in clinical pharmacy practice. Trial Registration: University of Maryland, Baltimore IRB; HP‐00112497.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Large Language ModelsInternal MedicinePharmacy Service, HospitalClinical Pharmacy Information SystemsHumansProspective StudiesGenerative Artificial IntelligenceArtificial Intelligence

Stichwörter der Autor:innen

artificial intelligencecomputerinternal medicineneural networkspharmacistspharmacy

Autor:innen und Institutionen

  • University of Maryland Medical Center, Baltimore, Maryland, USA.
  • University of Maryland School of Pharmacy, Baltimore, Maryland, USA.

ORCID-Profile: Samuel S Yang, Clement E Ng, Hyunuk Seung

Zitieren

APA

Yang, S. S., Ng, C. E., Seung, H., & Kelly, S. (2026). Comparison of Large Language Model‐Based Systems and Prompt Engineering for Internal Medicine Clinical Pharmacy Cases. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY, 9(7). https://doi.org/10.1002/jac5.70235

Vancouver

Yang SS, Ng CE, Seung H, Kelly S. Comparison of Large Language Model‐Based Systems and Prompt Engineering for Internal Medicine Clinical Pharmacy Cases. JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY. 2026;9(7). doi:10.1002/jac5.70235

BibTeX
@article{Yang2026,
  author = {Yang, Samuel S and Ng, Clement E and Seung, Hyunuk and Kelly, Sean},
  title = {Comparison of Large Language Model‐Based Systems and Prompt Engineering for Internal Medicine Clinical Pharmacy Cases},
  journal = {JACCP: JOURNAL OF THE AMERICAN COLLEGE OF CLINICAL PHARMACY},
  year = {2026},
  volume = {9},
  number = {7},
  doi = {10.1002/jac5.70235}
}
Kostenlos starten →