VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions
JAMIA Open · 26.12.2024 · 8 (1)
DOI: 10.1093/jamiaopen/ooaf005 ↗Kurz gefasst
Fachartikel in der Zeitschrift JAMIA Open, online erschienen am 19.02.2025. Verfasst von Yiming Li und 9 weiteren Autor:innen. Beteiligte Einrichtung: McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States. Die Arbeit wurde bisher 35-mal zitiert (Crossref) und nennt 71 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: NIAID NIH HHS, NIDDK NIH HHS, National Institute of Allergy and Infectious Diseases.
Abstract (Originalsprache)
Objective
Human Papillomavirus (HPV) vaccine is an effective measure to prevent and control the diseases caused by HPV. However, widespread misinformation and vaccine hesitancy remain significant barriers to its uptake. This study focuses on the development of VaxBot-HPV, a chatbot aimed at improving health literacy and promoting vaccination uptake by providing information and answering questions about the HPV vaccine.
Methods
We constructed the knowledge base (KB) for VaxBot-HPV, which consists of 451 documents from biomedical literature and web sources on the HPV vaccine. We extracted 202 question-answer pairs from the KB and 39 questions generated by GPT-4 for training and testing purposes. To comprehensively understand the capabilities and potential of GPT-based chatbots, 3 models were involved in this study: GPT-3.5, VaxBot-HPV, and GPT-4. The evaluation criteria included answer relevancy and faithfulness.
Results
VaxBot-HPV demonstrated superior performance in answer relevancy and faithfulness compared to baselines. For test questions in KB, it achieved an answer relevancy score of 0.85 and a faithfulness score of 0.97. Similarly, it attained scores of 0.85 for answer relevancy and 0.96 for faithfulness on GPT-generated questions.
Discussion
VaxBot-HPV demonstrates the effectiveness of fine-tuned large language models in healthcare, outperforming generic GPT models in accuracy and relevance. Fine-tuning mitigates hallucinations and misinformation, ensuring reliable information on HPV vaccination while allowing dynamic and tailored responses. The specific fine-tuning, which includes context in addition to question-answer pairs, enables VaxBot-HPV to provide explanations and reasoning behind its answers, enhancing transparency and user trust.
Conclusions
This study underscores the importance of leveraging large language models and fine-tuning techniques in the development of chatbots for healthcare applications, with implications for improving medical education and public health communication.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- McWilliams School of Biomedical Informatics, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States.
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Jacksonville, FL 32224, United States.
- Department of Biostatistics and Data Science, School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States.
- Department of Health Promotion and Behavioral Sciences, School of Public Health, The University of Texas Health Science Center at Houston, Houston, TX 77030, United States.
- Department of Biostatistics and Data Science, School of Public and Population Health, The University of Texas Medical Branch, Galveston, TX 77550, United States.
- Department of Internal Medicine, The University of Texas Medical Branch, Galveston, TX 77550, United States.
ORCID-Profile: Yiming Li, Lu Tang, Licong Cui
Förderung
- NIAID NIH HHS
- NIDDK NIH HHS
- National Institute of Allergy and Infectious Diseases
- National Institutes of Health
- National Institute of Diabetes and Digestive and Kidney Diseases
- CPRIT
Zitieren
APA
Li, Y., Li, J., Li, M., Yu, E., Rhee, D., Amith, M., Tang, L., Savas, L. S., Cui, L., & Tao, C. (2024). VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions. JAMIA Open, 8(1). https://doi.org/10.1093/jamiaopen/ooaf005
Vancouver
Li Y, Li J, Li M, Yu E, Rhee D, Amith M, et al. VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions. JAMIA Open. 2024;8(1). doi:10.1093/jamiaopen/ooaf005
BibTeX
@article{Li2024,
author = {Li, Yiming and Li, Jianfu and Li, Manqi and Yu, Evan and Rhee, Danniel and Amith, Muhammad and Tang, Lu and Savas, Lara S and Cui, Licong and Tao, Cui},
title = {VaxBot-HPV: a GPT-based chatbot for answering HPV vaccine-related questions},
journal = {JAMIA Open},
year = {2024},
volume = {8},
number = {1},
doi = {10.1093/jamiaopen/ooaf005}
}
EUVIMED