Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint)
JMIR Cancer · 2025
DOI: 10.2196/82971 ↗At a glance
Mixed-Methods-Studie in JMIR Cancer, 2025. Written by Agnik Saha and 4 co-authors. Institution: Department of Computer Science, Georgia State University, Atlanta, GA, United States. Major MEDLINE topics: Large Language Models, Breast Neoplasms, Uterine Cervical Neoplasms, Health Communication. 50 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Effective communication about breast and cervical cancers remains a public health challenge, with widespread misinformation and barriers to cancer-related language understanding. Large language models (LLMs) offer potential for scalable health communication, yet trade-offs between quality, safety, and accessibility of general-purpose and medical-domain LLMs remain underexplored.
Objective
This study aimed to propose a comprehensive evaluation framework and systematically assess the performance of LLMs in generating breast and cervical cancer information, with a focus on linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness.
Methods
This mixed methods evaluation study assessed outputs from 5 general-purpose and 3 medical LLMs using real-world breast and cervical cancer-related questions curated from publicly available medical datasets. LLM-generated responses were evaluated in a controlled offline setting. Primary outcomes included linguistic quality (fluency, coherence, and accuracy), safety and trustworthiness (toxicity, bias, and harm potential), and communication accessibility and affectiveness (readability, empathy, and clarity). Qualitative ratings were performed by domain experts, while quantitative metrics were compared across models. Statistical analyses included Welch ANOVA to detect differences in metric scores, Games-Howell tests for pairwise comparisons, and Hedges g to assess effect sizes.
Results
General-purpose LLMs, particularly Llama 3 and Gemma, demonstrated superior linguistic quality and affectiveness but often produced complex outputs that may limit accessibility. In contrast, medical LLMs (eg, MedAlpaca and BioMistral) generated simpler content suitable for broader audiences but scored lower in safety and empathy due to higher levels of hallucination, bias, and toxicity.
Conclusions
While LLMs show promise for improving digital cancer communication, our findings reveal a trade-off between domain specialization and overall communication quality and safety. Future development of health-focused LLMs should prioritize hybrid modeling strategies to enhance trust, clarity, and clinical relevance in patient-facing tools.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- Department of Computer Science, Georgia State University, Atlanta, GA, United States.
- Health Education and Promotion, College of Health and Human Performance, East Carolina University, Greenville, NC, United States.
- Morehouse School of Medicine, 720 Westview Drive, SW, Atlanta, GA, 30310-1496, United States, 1 404-756-8962.
- Institute for Insight at the J Mack Robinson College of Business, Georgia State University, Atlanta, GA, United States.
ORCID profiles: Agnik Saha, Victoria Churchill, Anny D Rodriguez, Ugur Kursuncu, Muhammed Y Idris
Cite
APA
Saha, A., Churchill, V., Rodriguez, A. D., Kursuncu, U., & Idris, M. Y. (2025). Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint). JMIR Cancer. https://doi.org/10.2196/82971
Vancouver
Saha A, Churchill V, Rodriguez AD, Kursuncu U, Idris MY. Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint). JMIR Cancer. 2025. doi:10.2196/82971
BibTeX
@article{Saha2025,
author = {Saha, Agnik and Churchill, Victoria and Rodriguez, Anny D and Kursuncu, Ugur and Idris, Muhammed Y},
title = {Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint)},
journal = {JMIR Cancer},
year = {2025},
doi = {10.2196/82971}
}
EUVIMED