EUVIMEDEUVIMED entdecken
Open Access

Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study

Sudeshna Das iD, Yao Ge iD, Yuting Guo iD, Swati Rajwal iD, JaMor Hairston iD, Jeanne Powell iD, Drew Walker iD, Snigdha Peddireddy iD, Sahithi Lakamana iD, Selen Bozkurt iD, Matthew Reyna iD, Reza Sameni iD und 10 weitere

Journal of Medical Internet Research · 06.01.2025 · 27 · e66220

DOI: 10.2196/66220 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 06.01.2025. Verfasst von Sudeshna Das und 21 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States. Hauptthemen laut MEDLINE: Social Media, Natural Language Processing, Proof of Concept Study. Die Arbeit wurde bisher 27-mal zitiert (Crossref) und nennt 19 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar. Gefördert von: NIDA NIH HHS.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The increasing use of social media to share lived and living experiences of substance use presents a unique opportunity to obtain information on side effects, use patterns, and opinions on novel psychoactive substances. However, due to the large volume of data, obtaining useful insights through natural language processing technologies such as large language models is challenging.

Objective

This paper aims to develop a retrieval-augmented generation (RAG) architecture for medical question answering pertaining to clinicians’ queries on emerging issues associated with health-related topics, using user-generated medical information on social media.

Methods

We proposed a two-layer RAG framework for query-focused answer generation and evaluated a proof of concept for the framework in the context of query-focused summary generation from social media forums, focusing on emerging drug-related information. Our modular framework generates individual summaries followed by an aggregated summary to answer medical queries from large amounts of user-generated social media data in an efficient manner. We compared the performance of a quantized large language model (Nous-Hermes-2-7B-DPO), deployable in low-resource settings, with GPT-4. For this proof-of-concept study, we used user-generated data from Reddit to answer clinicians’ questions on the use of xylazine and ketamine.

Results

Our framework achieves comparable median scores in terms of relevance, length, hallucination, coverage, and coherence when evaluated using GPT-4 and Nous-Hermes-2-7B-DPO, evaluated for 20 queries with 76 samples. There was no statistically significant difference between GPT-4 and Nous-Hermes-2-7B-DPO for coverage (Mann-Whitney U=733.0; n1=37; n2=39; P=.89 two-tailed), coherence (U=670.0; n1=37; n2=39; P=.49 two-tailed), relevance (U=662.0; n1=37; n2=39; P=.15 two-tailed), length (U=672.0; n1=37; n2=39; P=.55 two-tailed), and hallucination (U=859.0; n1=37; n2=39; P=.01 two-tailed). A statistically significant difference was noted for the Coleman-Liau Index (U=307.5; n1=20; n2=16; P

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Social MediaNatural Language ProcessingProof of Concept StudyHumansInformation Storage and Retrieval

Stichwörter der Autor:innen

GPTartificial intelligencelarge language modelsnatural language processingpsychoactive substanceretrieval-augmented generationsocial mediasubstance use

Autor:innen und Institutionen

  • Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States.
  • Department of Computer Science and Informatics, Emory University, Atlanta, GA, United States.
  • Department of Behavioral, Social & Health Education Sciences, Rollins School of Public Health, Emory University, Atlanta, GA, United States.
  • Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, United States.
  • Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, United States.
  • Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States.

ORCID-Profile: Sudeshna Das, Yao Ge, Yuting Guo, Swati Rajwal, JaMor Hairston, Jeanne Powell, Drew Walker, Snigdha Peddireddy, Sahithi Lakamana, Selen Bozkurt, Matthew Reyna, Reza Sameni, Yunyu Xiao, Sangmi Kim, Rasheeta Chandler, Natalie Hernandez, Danielle Mowery, Rachel Wightman, Jennifer Love, Anthony Spadaro, Jeanmarie Perrone, Abeed Sarker

Förderung

  • NIDA NIH HHS

Zitieren

APA

Das, S., Ge, Y., Guo, Y., Rajwal, S., Hairston, J., Powell, J., Walker, D., Peddireddy, S., Lakamana, S., Bozkurt, S., Reyna, M., Sameni, R., Xiao, Y., Kim, S., Chandler, R., Hernandez, N., Mowery, D., Wightman, R., Love, J., …, & Sarker, A. (2025). Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study. Journal of Medical Internet Research, 27, e66220. https://doi.org/10.2196/66220

Vancouver

Das S, Ge Y, Guo Y, Rajwal S, Hairston J, Powell J, et al. Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study. Journal of Medical Internet Research. 2025;27:e66220. doi:10.2196/66220

BibTeX
@article{Das2025,
  author = {Das, Sudeshna and Ge, Yao and Guo, Yuting and Rajwal, Swati and Hairston, JaMor and Powell, Jeanne and Walker, Drew and Peddireddy, Snigdha and Lakamana, Sahithi and Bozkurt, Selen and Reyna, Matthew and Sameni, Reza and Xiao, Yunyu and Kim, Sangmi and Chandler, Rasheeta and Hernandez, Natalie and Mowery, Danielle and Wightman, Rachel and Love, Jennifer and Spadaro, Anthony and Perrone, Jeanmarie and Sarker, Abeed},
  title = {Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e66220},
  doi = {10.2196/66220}
}
Kostenlos starten →