Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study
JMIR Formative Research · 13.11.2026 · 10 · e92821-e92821
DOI: 10.2196/92821 ↗Kurz gefasst
Fachartikel in der Zeitschrift JMIR Formative Research, online erschienen am 13.11.2026. Verfasst von Rezaur Rashid und 3 weiteren Autor:innen. Beteiligte Einrichtung: Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, …. Hauptthemen laut MEDLINE: Large Language Models, Benchmarking, Social Media. Die Arbeit nennt 55 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
Pregnancy-related weight gain is both physiologically complex and socially charged, and online discussions often combine clinical information with implicit emotional distress. Although automated sentiment analysis could support research on pregnancy-related discourse, model performance in this context remains inadequately characterized relative to expert human judgment.
Objective
This study aimed to benchmark contemporary sentiment classification models on pregnancy-related weight discussions, with emphasis on agreement with expert consensus, interpretive ambiguity, task adaptation, and deployment across posts and comments.
Methods
We curated 15,619 Reddit posts and comments using a large language model (LLM)–guided keyword expansion strategy and established an adjudicated consensus reference set of 200 posts. We evaluated 10 models on a fixed held-out test set (N=150) using aligned text inputs, weighted performance metrics, and 1000-resample bootstrap CIs. RoBERTa (Robustly Optimized Bidirectional Encoder Representations from Transformers Approach) and BERTweet (Bidirectional Encoder Representations from Transformers for Tweets) were additionally evaluated in a 5-fold fine-tuning sensitivity analysis. Gemini-3-Flash (Google DeepMind [Google LLC]) and Generative Pretrained Transformer (GPT)–OSS (OpenAI) were then applied to 400 unlabeled posts and 15,019 comments with intermodel agreement assessment. To expand expert-labeled coverage, a separate 106-post annotation extension was used for secondary evaluation of the 2 models carried forward to post-level corpus inference, and cross-source recoverability of the API-derived retrieval was assessed using Arctic Shift (Arthur Heitmann).
Results
The 2 primary experts (FAK and BMW) achieved 59.0% preadjudication agreement (Cohen κ=0.344). In the fixed held-out comparison, Gemini-3-Flash had the numerically highest weighted F 1 -score of 0.662 (κ=0.478), followed by GPT-OSS ( F 1 -score=0.645; κ=0.436), with overlapping CIs across leading models. Under the harmonized direct-argmax decision rule with minimally processed text, BERTweet achieved an F 1 -score of 0.623 (κ=0.369), and RoBERTa achieved an F 1 -score of 0.605 (κ=0.339). In the separate fine-tuning sensitivity analysis, the mean cross-validated F 1 -score was 0.708 (SD 0.087) for RoBERTa and 0.616 (SD 0.079) for BERTweet. Among Gemini-3-Flash errors, 55.8% (29/52) occurred in posts on which the 2 primary experts disagreed. In the 106-post annotation extension, primary-expert agreement was 64.2% (68/106; κ=0.367); Gemini-3-Flash and GPT-OSS achieved weighted F 1 -score values of 0.633 and 0.680, respectively, and both identified 56 of 58 expert-negative posts in an exploratory human-review prioritization analysis. All 714 API-retrieved post IDs were independently recoverable through Arctic Shift. Intermodel agreement was 92.3% for posts (κ=0.797; n=400 valid pairs) and 46.3% for comments (κ=0.102; n=1500).
Conclusions
No single model architecture was uniformly superior, and performance depended on the adaptation regime, annotation ambiguity, and discourse type. These findings define important boundary conditions for sentiment analysis of naturalistic maternal-health discourse: broad polarity classification can support corpus prioritization and hypothesis generation but is insufficient as a stand-alone measure of patient experience.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Center for Biomedical Informatics, Department of Pediatrics, College of Medicine, University of Tennessee Health Science Center, 50 N Dunlap Street, Memphis, TN, 38103, United States, +1 (901) 287-5863.
Zitieren
APA
Rashid, R., White, B. M., Kumsa, F. A., & Shaban-Nejad, A. (2026). Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study. JMIR Formative Research, 10, e92821-e92821. https://doi.org/10.2196/92821
Vancouver
Rashid R, White BM, Kumsa FA, Shaban-Nejad A. Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study. JMIR Formative Research. 2026;10:e92821-e92821. doi:10.2196/92821
BibTeX
@article{Rashid2026,
author = {Rashid, Rezaur and White, Brianna M and Kumsa, Fekede Asefa and Shaban-Nejad, Arash},
title = {Benchmarking Large Language Models for Sentiment Analysis in Pregnancy-Related Weight Discussions on Reddit: Human Performance-Aligned Comparative Evaluation Study},
journal = {JMIR Formative Research},
year = {2026},
volume = {10},
pages = {e92821--e92821},
doi = {10.2196/92821}
}
EUVIMED