EUVIMEDOpdag EUVIMED
Open access

Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study

Sudeshna Das iD, Yao Ge iD, Yuting Guo iD, Swati Rajwal iD, JaMor Hairston iD, Jeanne Powell iD, Drew Walker iD, Snigdha Peddireddy iD, Sahithi Lakamana iD, Selen Bozkurt iD, Matthew Reyna iD, Reza Sameni iD and 10 more

Journal of Medical Internet Research · 2025-01-06 · 27 · e66220

DOI: 10.2196/66220 ↗

At a glance

Journal article in Journal of Medical Internet Research, published online on 2025-01-06. Written by Sudeshna Das and 21 co-authors. Institution: Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States. Major MEDLINE topics: Social Media, Natural Language Processing, Proof of Concept Study. Cited 27 times so far (Crossref); 19 references. The full text is freely available in PubMed Central. Funded by: NIDA NIH HHS.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

The increasing use of social media to share lived and living experiences of substance use presents a unique opportunity to obtain information on side effects, use patterns, and opinions on novel psychoactive substances. However, due to the large volume of data, obtaining useful insights through natural language processing technologies such as large language models is challenging.

Objective

This paper aims to develop a retrieval-augmented generation (RAG) architecture for medical question answering pertaining to clinicians’ queries on emerging issues associated with health-related topics, using user-generated medical information on social media.

Methods

We proposed a two-layer RAG framework for query-focused answer generation and evaluated a proof of concept for the framework in the context of query-focused summary generation from social media forums, focusing on emerging drug-related information. Our modular framework generates individual summaries followed by an aggregated summary to answer medical queries from large amounts of user-generated social media data in an efficient manner. We compared the performance of a quantized large language model (Nous-Hermes-2-7B-DPO), deployable in low-resource settings, with GPT-4. For this proof-of-concept study, we used user-generated data from Reddit to answer clinicians’ questions on the use of xylazine and ketamine.

Results

Our framework achieves comparable median scores in terms of relevance, length, hallucination, coverage, and coherence when evaluated using GPT-4 and Nous-Hermes-2-7B-DPO, evaluated for 20 queries with 76 samples. There was no statistically significant difference between GPT-4 and Nous-Hermes-2-7B-DPO for coverage (Mann-Whitney U=733.0; n1=37; n2=39; P=.89 two-tailed), coherence (U=670.0; n1=37; n2=39; P=.49 two-tailed), relevance (U=662.0; n1=37; n2=39; P=.15 two-tailed), length (U=672.0; n1=37; n2=39; P=.55 two-tailed), and hallucination (U=859.0; n1=37; n2=39; P=.01 two-tailed). A statistically significant difference was noted for the Coleman-Liau Index (U=307.5; n1=20; n2=16; P

Subjects

MeSH terms (MEDLINE)

Social MediaNatural Language ProcessingProof of Concept StudyHumansInformation Storage and Retrieval

Author keywords

GPTartificial intelligencelarge language modelsnatural language processingpsychoactive substanceretrieval-augmented generationsocial mediasubstance use

Authors and institutions

  • Department of Biomedical Informatics, School of Medicine, Emory University, Atlanta, GA, United States.
  • Department of Computer Science and Informatics, Emory University, Atlanta, GA, United States.
  • Department of Behavioral, Social & Health Education Sciences, Rollins School of Public Health, Emory University, Atlanta, GA, United States.
  • Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, United States.
  • Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, United States.
  • Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA, United States.

ORCID profiles: Sudeshna Das, Yao Ge, Yuting Guo, Swati Rajwal, JaMor Hairston, Jeanne Powell, Drew Walker, Snigdha Peddireddy, Sahithi Lakamana, Selen Bozkurt, Matthew Reyna, Reza Sameni, Yunyu Xiao, Sangmi Kim, Rasheeta Chandler, Natalie Hernandez, Danielle Mowery, Rachel Wightman, Jennifer Love, Anthony Spadaro, Jeanmarie Perrone, Abeed Sarker

Funding

  • NIDA NIH HHS

Cite

APA

Das, S., Ge, Y., Guo, Y., Rajwal, S., Hairston, J., Powell, J., Walker, D., Peddireddy, S., Lakamana, S., Bozkurt, S., Reyna, M., Sameni, R., Xiao, Y., Kim, S., Chandler, R., Hernandez, N., Mowery, D., Wightman, R., Love, J., …, & Sarker, A. (2025). Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study. Journal of Medical Internet Research, 27, e66220. https://doi.org/10.2196/66220

Vancouver

Das S, Ge Y, Guo Y, Rajwal S, Hairston J, Powell J, et al. Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study. Journal of Medical Internet Research. 2025;27:e66220. doi:10.2196/66220

BibTeX
@article{Das2025,
  author = {Das, Sudeshna and Ge, Yao and Guo, Yuting and Rajwal, Swati and Hairston, JaMor and Powell, Jeanne and Walker, Drew and Peddireddy, Snigdha and Lakamana, Sahithi and Bozkurt, Selen and Reyna, Matthew and Sameni, Reza and Xiao, Yunyu and Kim, Sangmi and Chandler, Rasheeta and Hernandez, Natalie and Mowery, Danielle and Wightman, Rachel and Love, Jennifer and Spadaro, Anthony and Perrone, Jeanmarie and Sarker, Abeed},
  title = {Two-Layer Retrieval-Augmented Generation Framework for Low-Resource Medical Question Answering Using Reddit Data: Proof-of-Concept Study},
  journal = {Journal of Medical Internet Research},
  year = {2025},
  volume = {27},
  pages = {e66220},
  doi = {10.2196/66220}
}
Start for free →