Evaluating large language models for health-related text classification tasks with public social media data
Journal of the American Medical Informatics Association · 01.10.2024 · 31 (10) · 2181-2189
DOI: 10.1093/jamia/ocae210 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the American Medical Informatics Association, online erschienen am 09.08.2024. Verfasst von Yuting Guo und 3 weiteren Autor:innen. Beteiligte Einrichtung: Department of Biomedical Informatics, Emory University, Atlanta, GA 30322, United States. Hauptthemen laut MEDLINE: Natural Language Processing, Social Media, Support Vector Machine. Die Arbeit wurde bisher 73-mal zitiert (Crossref) und nennt 42 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: NIDA NIH HHS, NIH, NIH HHS.
Abstract (Originalsprache)
Objectives
Large language models (LLMs) have demonstrated remarkable success in natural language processing (NLP) tasks. This study aimed to evaluate their performances on social media-based health-related text classification tasks. Materials and Methods We benchmarked 1 Support Vector Machine (SVM), 3 supervised pretrained language models (PLMs), and 2 LLMs-based classifiers across 6 text classification tasks. We developed 3 approaches for leveraging LLMs: employing LLMs as zero-shot classifiers, using LLMs as data annotators, and utilizing LLMs with few-shot examples for data augmentation.
Results
Across all tasks, the mean (SD) F1 score differences for RoBERTa, BERTweet, and SocBERT trained on human-annotated data were 0.24 (±0.10), 0.25 (±0.11), and 0.23 (±0.11), respectively, compared to those trained on the data annotated using GPT3.5, and were 0.16 (±0.07), 0.16 (±0.08), and 0.14 (±0.08) using GPT4, respectively. The GPT3.5 and GPT4 zero-shot classifiers outperformed SVMs in a single task and in 5 out of 6 tasks, respectively. When leveraging LLMs for data augmentation, the RoBERTa models trained on GPT4-augmented data demonstrated superior or comparable performance compared to those trained on human-annotated data alone.
Discussion
The results revealed that using LLM-annotated data only for training supervised classification models was ineffective. However, employing the LLM as a zero-shot classifier exhibited the potential to outperform traditional SVM models and achieved a higher recall than the advanced transformer-based model RoBERTa. Additionally, our results indicated that utilizing GPT3.5 for data augmentation could potentially harm model performance. In contrast, data augmentation with GPT4 demonstrated improved model performances, showcasing the potential of LLMs in reducing the need for extensive training data.
Conclusions
By leveraging the data augmentation strategy, we can harness the power of LLMs to develop smaller, more effective domain-specific NLP models. Using LLM-annotated data without human guidance for training lightweight supervised classification models is an ineffective strategy. However, LLM, as a zero-shot classifier, shows promise in excluding false negatives and potentially reducing the human effort required for data annotation.
Schlagwörter
MeSH-Schlagwörter (MEDLINE)
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Biomedical Informatics, Emory University, Atlanta, GA 30322, United States.
- Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA 30332, United States.
- Department of Biomedical Informatics, Vanderbilt University, Nashville, TN 37235, United States.
ORCID-Profile: Abeed Sarker
Förderung
- NIDA NIH HHS
- NIH
- NIH HHS
- NIDA
- National Institute on Drug Abuse
- National Institutes of Health
Zitieren
APA
Guo, Y., Ovadje, A., Al-Garadi, M. A., & Sarker, A. (2024). Evaluating large language models for health-related text classification tasks with public social media data. Journal of the American Medical Informatics Association, 31(10), 2181-2189. https://doi.org/10.1093/jamia/ocae210
Vancouver
Guo Y, Ovadje A, Al-Garadi MA, Sarker A. Evaluating large language models for health-related text classification tasks with public social media data. Journal of the American Medical Informatics Association. 2024;31(10):2181-2189. doi:10.1093/jamia/ocae210
BibTeX
@article{Guo2024,
author = {Guo, Yuting and Ovadje, Anthony and Al-Garadi, Mohammed Ali and Sarker, Abeed},
title = {Evaluating large language models for health-related text classification tasks with public social media data},
journal = {Journal of the American Medical Informatics Association},
year = {2024},
volume = {31},
number = {10},
pages = {2181--2189},
doi = {10.1093/jamia/ocae210}
}
EUVIMED