Assessing Large Language Models in Building a Structured Dataset From AskDocs Subreddit Data: Methodological Study
Journal of Medical Internet Research · 2025
DOI: 10.2196/74094 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Medical Internet Research, 2025. Verfasst von Quinn Snell und 5 weiteren Autor:innen. Beteiligte Einrichtung: Brigham Young University, 3361 TMCB, Provo, UT, 84602, United States, 1 8014225098. Hauptthemen laut MEDLINE: Language, Social Media. Die Arbeit nennt 31 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.
Abstract (Originalsprache)
Background
In an era marked by a growing reliance on digital platforms for health care consultation, the subreddit r/AskDocs has emerged as a pivotal forum. However, the vast, unstructured nature of forum data presents a formidable challenge; the extraction and meaningful analysis of such data require advanced tools that can navigate the complexities of language and context inherent in user-generated content. The emergence of large language models (LLMs) offers new tools for the extraction of health-related content from unstructured text found in social media platforms such as Reddit.
Objective
This methodological study aimed to evaluate the use of LLMs to systematically transform the rich, unstructured textual data from the AskDocs subreddit into a structured dataset, an approach that aligns more closely with human cognitive processes than traditional data extraction methods.
Methods
Human annotators and LLMs were used to extract data from 2800 randomly sampled r/AskDocs subreddit posts. For human annotation, at least 2 medical students extracted demographic information, type of inquiry (diagnosis, symptom, or treatment), proxy relationship, chronic condition, health care consultation status, and primary focus topic. For LLM data extraction, specially engineered prompts were created using JavaScript Object Notation and few-shot prompting. Prompts were used to query several state-of-the-art LLMs (eg, Llama 3, Genna, and GPT). Cohen κ was calculated across all human annotators, with this dataset serving as the gold standard for comparison with LLM data extraction. A high degree of human annotator reliability was observed for the coding of demographic information. Lower reliability was seen in coding the health-related content of the posts.
Results
The highest performance scores compared with the gold standard were achieved by Llama 3 70B with 7 few-shot prompt examples (average accuracy=87.4) and GPT-4 with 2 few-shot prompt examples (average accuracy=87.4) . Llama 3 70B excelled in coding health-related content while GPT-4 performed better coding demographic content from unstructured posts.
Conclusions
LLMs performed comparably with human annotators in extracting demographic and health-related information from the AskDocs subreddit unstructured posts. This study validates the use of LLMs for analyzing digital health care communications and highlights their potential as reliable tools for understanding online behaviors and interactions, shifting toward more sophisticated methodologies in digital research and practice.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Brigham Young University, 3361 TMCB, Provo, UT, 84602, United States, 1 8014225098.
- University of Nevada, Reno, Reno, NV, United States.
Zitieren
APA
Snell, Q., Westhoff, C., Westhoff, J., Low, E., Hanson, C. L., & Tass, E. S. N. (2025). Assessing Large Language Models in Building a Structured Dataset From AskDocs Subreddit Data: Methodological Study. Journal of Medical Internet Research. https://doi.org/10.2196/74094
Vancouver
Snell Q, Westhoff C, Westhoff J, Low E, Hanson CL, Tass ESN. Assessing Large Language Models in Building a Structured Dataset From AskDocs Subreddit Data: Methodological Study. Journal of Medical Internet Research. 2025. doi:10.2196/74094
BibTeX
@article{Snell2025,
author = {Snell, Quinn and Westhoff, Chase and Westhoff, John and Low, Ethan and Hanson, Carl L and Tass, E Shannon Neeley},
title = {Assessing Large Language Models in Building a Structured Dataset From AskDocs Subreddit Data: Methodological Study},
journal = {Journal of Medical Internet Research},
year = {2025},
doi = {10.2196/74094}
}
EUVIMED