EUVIMEDEUVIMED entdecken
Open Access

Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments

Chuan Hong iD, Anand Chowdhury iD, Anthony D Sorrentino, Haoyuan Wang, Monica Agrawal, Armando Bedoya iD, Sophia Bessias, Nicoleta J Economou-Zavlanos iD, Ian Wong, Christian Pean, Fan Li, Kathryn I Pollak iD und 2 weitere

Journal of the American Medical Informatics Association · 01.04.2025 · 32 (4) · 626-637

DOI: 10.1093/jamia/ocaf023 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 10.03.2025. Verfasst von Chuan Hong und 13 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27710, United States. Hauptthemen laut MEDLINE: Artificial Intelligence, Natural Language Processing, Unified Medical Language System. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 86 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objectives

Large language models (LLMs) are increasingly utilized in healthcare, transforming medical practice through advanced language processing capabilities. However, the evaluation of LLMs predominantly relies on human qualitative assessment, which is time-consuming, resource-intensive, and may be subject to variability and bias. There is a pressing need for quantitative metrics to enable scalable, objective, and efficient evaluation. Materials and Methods We propose a unified evaluation framework that bridges qualitative and quantitative methods to assess LLM performance in healthcare settings. This framework maps evaluation aspects—such as linguistic quality, efficiency, content integrity, trustworthiness, and usefulness—to both qualitative assessments and quantitative metrics. We apply our approach to empirically evaluate the Epic In-Basket feature, which uses LLM to generate patient message replies.

Results

The empirical evaluation demonstrates that while Artificial Intelligence (AI)-generated replies exhibit high fluency, clarity, and minimal toxicity, they face challenges with coherence and completeness. Clinicians’ manual decision to use AI-generated drafts correlates strongly with quantitative metrics, suggesting that quantitative metrics have the potential to reduce human effort in the evaluation process and make it more scalable.

Discussion

Our study highlights the potential of a unified evaluation framework that integrates qualitative and quantitative methods, enabling scalable and systematic assessments of LLMs in healthcare. Automated metrics streamline evaluation and monitoring processes, but their effective use depends on alignment with human judgment, particularly for aspects requiring contextual interpretation. As LLM applications expand, refining evaluation strategies and fostering interdisciplinary collaboration will be critical to maintaining high standards of accuracy, ethics, and regulatory compliance.

Conclusion

Our unified evaluation framework bridges the gap between qualitative human assessments and automated quantitative metrics, enhancing the reliability and scalability of LLM evaluations in healthcare. While automated quantitative evaluations are not ready to fully replace qualitative human evaluations, they can be used to enhance the process and, with relevant benchmarks derived from the unified framework proposed here, they can be applied to LLM monitoring and evaluation of updated versions of the original technology evaluated using qualitative human standards.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Artificial IntelligenceNatural Language ProcessingUnified Medical Language SystemHumansLarge Language Models

Stichwörter der Autor:innen

generative AIhealthcarelarge language modelqualitative evaluationquantitative evaluation

Autor:innen und Institutionen

  • Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, NC 27710, United States.
  • Department of Medicine, Duke University School of Medicine, Durham, NC 27710, United States.
  • Duke Clinical and Translational Science Institute, Duke University School of Medicine, Durham, NC 27710, United States.
  • Department of Statistical Science, Duke University, Durham, NC 27710, United States.
  • Cancer Prevention and Control Research Program, Duke Cancer Institute, Durham, NC 27710, United States.
  • Department of Population Health Sciences, Duke University School of Medicine, Durham, NC 27710, United States.

ORCID-Profile: Chuan Hong, Anand Chowdhury, Armando Bedoya, Nicoleta J Economou-Zavlanos, Kathryn I Pollak, Eric G Poon, Michael J Pencina

Zitieren

APA

Hong, C., Chowdhury, A., Sorrentino, A. D., Wang, H., Agrawal, M., Bedoya, A., Bessias, S., Economou-Zavlanos, N. J., Wong, I., Pean, C., Li, F., Pollak, K. I., Poon, E. G., & Pencina, M. J. (2025). Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments. Journal of the American Medical Informatics Association, 32(4), 626-637. https://doi.org/10.1093/jamia/ocaf023

Vancouver

Hong C, Chowdhury A, Sorrentino AD, Wang H, Agrawal M, Bedoya A, et al. Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments. Journal of the American Medical Informatics Association. 2025;32(4):626-637. doi:10.1093/jamia/ocaf023

BibTeX
@article{Hong2025,
  author = {Hong, Chuan and Chowdhury, Anand and Sorrentino, Anthony D and Wang, Haoyuan and Agrawal, Monica and Bedoya, Armando and Bessias, Sophia and Economou-Zavlanos, Nicoleta J and Wong, Ian and Pean, Christian and Li, Fan and Pollak, Kathryn I and Poon, Eric G and Pencina, Michael J},
  title = {Application of unified health large language model evaluation framework to In-Basket message replies: bridging qualitative and quantitative assessments},
  journal = {Journal of the American Medical Informatics Association},
  year = {2025},
  volume = {32},
  number = {4},
  pages = {626--637},
  doi = {10.1093/jamia/ocaf023}
}
Kostenlos starten →