EUVIMEDSkopri EUVIMED
Mixed-Methods-StudieOpen access

Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint)

Agnik Saha iD, Victoria Churchill iD, Anny D Rodriguez iD, Ugur Kursuncu iD, Muhammed Y Idris iD

JMIR Cancer · 2025

DOI: 10.2196/82971 ↗

At a glance

Mixed-Methods-Studie in JMIR Cancer, 2025. Written by Agnik Saha and 4 co-authors. Institution: Department of Computer Science, Georgia State University, Atlanta, GA, United States. Major MEDLINE topics: Large Language Models, Breast Neoplasms, Uterine Cervical Neoplasms, Health Communication. 50 references. The full text is freely available in PubMed Central.

AstrattSubjectsDettalji biblijografiċiCiteRelated studies

Abstract (original language)

Background

Effective communication about breast and cervical cancers remains a public health challenge, with widespread misinformation and barriers to cancer-related language understanding. Large language models (LLMs) offer potential for scalable health communication, yet trade-offs between quality, safety, and accessibility of general-purpose and medical-domain LLMs remain underexplored.

Objective

This study aimed to propose a comprehensive evaluation framework and systematically assess the performance of LLMs in generating breast and cervical cancer information, with a focus on linguistic quality, safety and trustworthiness, and communication accessibility and affectiveness.

Methods

This mixed methods evaluation study assessed outputs from 5 general-purpose and 3 medical LLMs using real-world breast and cervical cancer-related questions curated from publicly available medical datasets. LLM-generated responses were evaluated in a controlled offline setting. Primary outcomes included linguistic quality (fluency, coherence, and accuracy), safety and trustworthiness (toxicity, bias, and harm potential), and communication accessibility and affectiveness (readability, empathy, and clarity). Qualitative ratings were performed by domain experts, while quantitative metrics were compared across models. Statistical analyses included Welch ANOVA to detect differences in metric scores, Games-Howell tests for pairwise comparisons, and Hedges g to assess effect sizes.

Results

General-purpose LLMs, particularly Llama 3 and Gemma, demonstrated superior linguistic quality and affectiveness but often produced complex outputs that may limit accessibility. In contrast, medical LLMs (eg, MedAlpaca and BioMistral) generated simpler content suitable for broader audiences but scored lower in safety and empathy due to higher levels of hallucination, bias, and toxicity.

Conclusions

While LLMs show promise for improving digital cancer communication, our findings reveal a trade-off between domain specialization and overall communication quality and safety. Future development of health-focused LLMs should prioritize hybrid modeling strategies to enhance trust, clarity, and clinical relevance in patient-facing tools.

Subjects

MeSH terms (MEDLINE)

Large Language ModelsBreast NeoplasmsUterine Cervical NeoplasmsHealth CommunicationHumansFemale

Author keywords

artificial intelligencehealth communicationlarge language modelsmedical informaticsnatural language processing

Authors and institutions

  • Department of Computer Science, Georgia State University, Atlanta, GA, United States.
  • Health Education and Promotion, College of Health and Human Performance, East Carolina University, Greenville, NC, United States.
  • Morehouse School of Medicine, 720 Westview Drive, SW, Atlanta, GA, 30310-1496, United States, 1 404-756-8962.
  • Institute for Insight at the J Mack Robinson College of Business, Georgia State University, Atlanta, GA, United States.

ORCID profiles: Agnik Saha, Victoria Churchill, Anny D Rodriguez, Ugur Kursuncu, Muhammed Y Idris

Cite

APA

Saha, A., Churchill, V., Rodriguez, A. D., Kursuncu, U., & Idris, M. Y. (2025). Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint). JMIR Cancer. https://doi.org/10.2196/82971

Vancouver

Saha A, Churchill V, Rodriguez AD, Kursuncu U, Idris MY. Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint). JMIR Cancer. 2025. doi:10.2196/82971

BibTeX
@article{Saha2025,
  author = {Saha, Agnik and Churchill, Victoria and Rodriguez, Anny D and Kursuncu, Ugur and Idris, Muhammed Y},
  title = {Large Language Models for Breast and Cervical Cancer Communication: Mixed-methods Evaluation of Linguistic Quality, Safety, and Accessibility (Preprint)},
  journal = {JMIR Cancer},
  year = {2025},
  doi = {10.2196/82971}
}
Start for free →