Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model
Mayo Clinic Proceedings: Digital Health · 2024-12 · 2 (4) · 632-644
DOI: 10.1016/j.mcpdig.2024.09.006 ↗Kurz gefasst
Fachartikel in der Zeitschrift Mayo Clinic Proceedings: Digital Health, 2024-12. Verfasst von Zilma Silveira Nogueira Reis und 12 weiteren Autor:innen. Beteiligte Einrichtung: Health Informatics Center, Faculty of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Gates Foundation, Bill & Melinda Gates Foundation, National Council for Scientific and Technological Development.
Abstract (Originalsprache)
Objective
To assess the support of large language models (LLMs) in generating clearer and more personalized medication instructions to enhance e-prescription. Patients and Methods: We established patient-centered guidelines for adequate, acceptable, and personalized directions to enhance e-prescription. A dataset comprising 104 outpatient scenarios, with an array of medications, administration routes, and patient conditions, was developed following the Brazilian national e-prescribing standard. Three prompts were submitted to a closed-source LLM. The first prompt involved a generic command, the second one was calibrated for content enhancement and personalization, and the third one requested bias mitigation. The third prompt was submitted to an open-source LLM. Outputs were assessed using automated metrics and human evaluation. We conducted the study between March 1, 2024 and September 10, 2024.
Results
Adequacy scores of our closed-source LLM's output showed the third prompt outperforming the first and second one. Full and partial acceptability was achieved in 94.3% of texts with the third prompt. Personalization was rated highly, especially with the second and third prompts. The 2 LLMs showed similar adequacy results. Lack of scientific evidence and factual errors were infrequent and unrelated to a particular prompt or LLM. The frequency of hallucinations was different for each LLM and concerned prescriptions issued upon symptom manifestation and medications requiring dosage adjustment or involving intermittent use. Gender bias was found in our closed-source LLM's output for the first and second prompts, with the third one being bias-free. The second LLM's output was bias-free.
Conclusion
This study demonstrates the potential of LLM-supported generation to produce prescription directions and improve communication between health professionals and patients within the e-prescribing system.
Autor:innen und Institutionen
- Health Informatics Center, Faculty of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
- Arts Faculty, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
- Department of Pediatrics, Faculty of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
- Department of Design Technology, School of Architecture, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
- Department of Obstetrics and Gynecology, Faculty of Medical Sciences, Universidade Estadual de Campinas, Campinas, São Paulo, Brazil.
- Departamento de Medicina da Comunidade Informação e Decisão em Saúde, National Secretary of Primary Care of the Brazilian Ministry of Health, Brasília, Brazil.
Förderung
- Gates Foundation
- Bill & Melinda Gates Foundation
- National Council for Scientific and Technological Development
Zitieren
APA
Reis, Z. S. N., Pagano, A. S., Ramos de Oliveira, I. J., Dias, C. D. S., Lage, E. M., Mineiro, E. F., Varella Pereira, G. M., de Carvalho Gomes, I., Basilio, V. A., Cruz-Correia, R. J., de Jesus, D. D. R., de Souza Júnior, A. P., & da Rocha, L. C. D. (2024). Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model. Mayo Clinic Proceedings: Digital Health, 2(4), 632-644. https://doi.org/10.1016/j.mcpdig.2024.09.006
Vancouver
Reis ZSN, Pagano AS, Ramos de Oliveira IJ, Dias CDS, Lage EM, Mineiro EF, et al. Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model. Mayo Clinic Proceedings: Digital Health. 2024;2(4):632-644. doi:10.1016/j.mcpdig.2024.09.006
BibTeX
@article{Reis2024,
author = {Reis, Zilma Silveira Nogueira and Pagano, Adriana Silvina and Ramos de Oliveira, Isaias Jose and Dias, Cristiane Dos Santos and Lage, Eura Martins and Mineiro, Erico Franco and Varella Pereira, Glaucia Miranda and de Carvalho Gomes, Igor and Basilio, Vinicius Araujo and Cruz-Correia, Ricardo João and de Jesus, Davi Dos Reis and de Souza Júnior, Antônio Pereira and da Rocha, Leonardo Chaves Dutra},
title = {Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model},
journal = {Mayo Clinic Proceedings: Digital Health},
year = {2024},
volume = {2},
number = {4},
pages = {632--644},
doi = {10.1016/j.mcpdig.2024.09.006}
}
EUVIMED