EUVIMEDEUVIMED entdecken
Open Access

Menstrual Health Education Using a Specialized Large Language Model in India: Development and Evaluation Study of MenstLLaMA

Prottay Kumar Adhikary, Isha Motiyani, Gayatri Oke, Maithili Joshi, Kanupriya Pathak, Salam Michael Singh, Tanmoy Chakraborty

Journal of Medical Internet Research · 2025

DOI: 10.2196/71977 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Prottay Kumar Adhikary und 6 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electrical Engineering, Indian Institute of Technology Delhi, Room: 3B-7 (Block III 3rd Floor), Hauz Khas, New Delhi, 110016, India, 91…. Hauptthemen laut MEDLINE: Menstruation, Health Education, Language. Die Arbeit wurde bisher 6-mal zitiert (Crossref) und nennt 45 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The quality and accessibility of menstrual health education (MHE) in low- and middle-income countries, including India, remain inadequate due to persistent challenges (eg, poverty, social stigma, and gender inequality). While community-driven initiatives have sought to raise awareness, artificial intelligence offers a scalable and efficient solution for disseminating accurate information. However, existing general-purpose large language models (LLMs) are often ill-suited for this task, tending to exhibit low accuracy, cultural insensitivity, and overly complex responses. To address these limitations, we developed MenstLLaMA—a specialized LLM tailored to the Indian context and designed to deliver MHE empathetically, supportively, and accessibly.

Objective

We aimed to develop and evaluate MenstLLaMA—a specialized LLM tailored to deliver accurate, culturally sensitive MHE—and assess its effectiveness in comparison to existing general-purpose models.

Methods

We curated MENST—a novel, domain-specific dataset comprising 23,820 question-answer pairs aggregated from medical websites, government portals, and health education resources. This dataset was systematically annotated with metadata capturing age groups, regions, topics, and sociocultural contexts. MenstLLaMA was developed by fine-tuning Meta-LLaMA-3-8B-Instruct, using parameter-efficient fine-tuning with low-rank adaptation to achieve domain alignment while minimizing computational overhead. We benchmarked MenstLLaMA against 9 state-of-the-art general-purpose LLMs, including GPT-4o, Claude-3, Gemini 1.5 Pro, and Mistral. The evaluation followed a multilayered framework: (1) automatic evaluation using standard natural language processing metrics (BLEU [Bilingual Evaluation Understudy], METEOR [Metric for Evaluation of Translation with Explicit Ordering], ROUGE-L [Recall-Oriented Understudy for Gisting Evaluation-Longest Common Subsequence], and BERTScore [Bidirectional Encoder Representations from Transformers Score]); (2) evaluation by clinical experts (N=18), who rated 200 expert-curated queries for accuracy and appropriateness; (3) medical practitioner interaction through the ISHA (Intelligent System for Menstrual Health Assistance) interactive chatbot, assessing qualitative dimensions (eg, relevance, understandability, preciseness, correctness, and context sensitivity ); and (4) a user study with volunteer participants (N=200), who evaluated MenstLLaMA in 15- to 20-minute randomized sessions, rating the system across 7 qualitative user satisfaction metrics.

Results

MenstLLaMA achieved the highest scores in BLEU (0.059) and BERTScore (0.911), outperforming GPT-4o (BLEU: 0.052, BERTScore: 0.896) and Claude-3 (BERTScore: 0.888). Clinical experts preferred MenstLLaMA’s responses over gold-standard answers in several culturally sensitive cases. In medical practitioners’ evaluations using the ISHA—the chat interface powered by MenstLLaMA—the model scored 3.5 in relevance , 3.6 in understandability , 3.1/5 in preciseness , 3.5/5 in correctness , and 4.0/5 in context sensitivity . User evaluations indicated even stronger results, with ratings of 4.7/5 for understandability , 4.3/5 for relevance , 4.28/5 for preciseness , 4.1/5 for correctness , 4.6/5 for tone , 4.2/5 for flow , and 3.9/5 for context sensitivity .

Conclusions

MenstLLaMA demonstrates exceptional accuracy, empathy, and user satisfaction within the domain of MHE, bridging critical gaps left by general-purpose LLMs. Its potential for integration into broader health education platforms positions it as a transformative tool for menstrual well-being. Future research could explore its long-term impact on public perception and menstrual hygiene practices, while expanding demographic representation, enhancing context sensitivity, and integrating multimodal and voice-based interactions to improve accessibility across diverse user groups.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

MenstruationHealth EducationLanguageHumansIndiaFemaleAdultAdolescentLarge Language Models

Stichwörter der Autor:innen

artificial intelligencecultural sensitivitydigital healthhealth equitylarge language modelmenstrual health education

Autor:innen und Institutionen

  • Department of Electrical Engineering, Indian Institute of Technology Delhi, Room: 3B-7 (Block III 3rd Floor), Hauz Khas, New Delhi, 110016, India, 91 26591076 ext 011.
  • Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India.

Zitieren

APA

Adhikary, P. K., Motiyani, I., Oke, G., Joshi, M., Pathak, K., Singh, S. M., & Chakraborty, T. (2025). Menstrual Health Education Using a Specialized Large Language Model in India: Development and Evaluation Study of MenstLLaMA. Journal of Medical Internet Research. https://doi.org/10.2196/71977

Vancouver

Adhikary PK, Motiyani I, Oke G, Joshi M, Pathak K, Singh SM, et al. Menstrual Health Education Using a Specialized Large Language Model in India: Development and Evaluation Study of MenstLLaMA. Journal of Medical Internet Research. 2025. doi:10.2196/71977

BibTeX
@article{Adhikary2025,
  author = {Adhikary, Prottay Kumar and Motiyani, Isha and Oke, Gayatri and Joshi, Maithili and Pathak, Kanupriya and Singh, Salam Michael and Chakraborty, Tanmoy},
  title = {Menstrual Health Education Using a Specialized Large Language Model in India: Development and Evaluation Study of MenstLLaMA},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  doi = {10.2196/71977}
}
Kostenlos starten →