EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model

Zilma Silveira Nogueira Reis, Adriana Silvina Pagano, Isaias Jose Ramos de Oliveira, Cristiane Dos Santos Dias, Eura Martins Lage, Erico Franco Mineiro, Glaucia Miranda Varella Pereira, Igor de Carvalho Gomes, Vinicius Araujo Basilio, Ricardo João Cruz-Correia, Davi Dos Reis de Jesus, Antônio Pereira de Souza Júnior und 1 weitere

Mayo Clinic Proceedings: Digital Health · 2024-12 · 2 (4) · 632-644

DOI: 10.1016/j.mcpdig.2024.09.006 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Mayo Clinic Proceedings: Digital Health, 2024-12. Verfasst von Zilma Silveira Nogueira Reis und 12 weiteren Autor:innen. Beteiligte Einrichtung: Health Informatics Center, Faculty of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 22 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Gates Foundation, Bill & Melinda Gates Foundation, National Council for Scientific and Technological Development.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Objective

To assess the support of large language models (LLMs) in generating clearer and more personalized medication instructions to enhance e-prescription. Patients and Methods: We established patient-centered guidelines for adequate, acceptable, and personalized directions to enhance e-prescription. A dataset comprising 104 outpatient scenarios, with an array of medications, administration routes, and patient conditions, was developed following the Brazilian national e-prescribing standard. Three prompts were submitted to a closed-source LLM. The first prompt involved a generic command, the second one was calibrated for content enhancement and personalization, and the third one requested bias mitigation. The third prompt was submitted to an open-source LLM. Outputs were assessed using automated metrics and human evaluation. We conducted the study between March 1, 2024 and September 10, 2024.

Results

Adequacy scores of our closed-source LLM's output showed the third prompt outperforming the first and second one. Full and partial acceptability was achieved in 94.3% of texts with the third prompt. Personalization was rated highly, especially with the second and third prompts. The 2 LLMs showed similar adequacy results. Lack of scientific evidence and factual errors were infrequent and unrelated to a particular prompt or LLM. The frequency of hallucinations was different for each LLM and concerned prescriptions issued upon symptom manifestation and medications requiring dosage adjustment or involving intermittent use. Gender bias was found in our closed-source LLM's output for the first and second prompts, with the third one being bias-free. The second LLM's output was bias-free.

Conclusion

This study demonstrates the potential of LLM-supported generation to produce prescription directions and improve communication between health professionals and patients within the e-prescribing system.

Autor:innen und Institutionen

  • Health Informatics Center, Faculty of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
  • Arts Faculty, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
  • Department of Pediatrics, Faculty of Medicine, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
  • Department of Design Technology, School of Architecture, Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil.
  • Department of Obstetrics and Gynecology, Faculty of Medical Sciences, Universidade Estadual de Campinas, Campinas, São Paulo, Brazil.
  • Departamento de Medicina da Comunidade Informação e Decisão em Saúde, National Secretary of Primary Care of the Brazilian Ministry of Health, Brasília, Brazil.

Förderung

  • Gates Foundation
  • Bill & Melinda Gates Foundation
  • National Council for Scientific and Technological Development

Zitieren

APA

Reis, Z. S. N., Pagano, A. S., Ramos de Oliveira, I. J., Dias, C. D. S., Lage, E. M., Mineiro, E. F., Varella Pereira, G. M., de Carvalho Gomes, I., Basilio, V. A., Cruz-Correia, R. J., de Jesus, D. D. R., de Souza Júnior, A. P., & da Rocha, L. C. D. (2024). Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model. Mayo Clinic Proceedings: Digital Health, 2(4), 632-644. https://doi.org/10.1016/j.mcpdig.2024.09.006

Vancouver

Reis ZSN, Pagano AS, Ramos de Oliveira IJ, Dias CDS, Lage EM, Mineiro EF, et al. Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model. Mayo Clinic Proceedings: Digital Health. 2024;2(4):632-644. doi:10.1016/j.mcpdig.2024.09.006

BibTeX
@article{Reis2024,
  author = {Reis, Zilma Silveira Nogueira and Pagano, Adriana Silvina and Ramos de Oliveira, Isaias Jose and Dias, Cristiane Dos Santos and Lage, Eura Martins and Mineiro, Erico Franco and Varella Pereira, Glaucia Miranda and de Carvalho Gomes, Igor and Basilio, Vinicius Araujo and Cruz-Correia, Ricardo João and de Jesus, Davi Dos Reis and de Souza Júnior, Antônio Pereira and da Rocha, Leonardo Chaves Dutra},
  title = {Evaluating Large Language Model–Supported Instructions for Medication Use: First Steps Toward a Comprehensive Model},
  journal = {Mayo Clinic Proceedings: Digital Health},
  year = {2024},
  volume = {2},
  number = {4},
  pages = {632--644},
  doi = {10.1016/j.mcpdig.2024.09.006}
}
Kostenlos starten →