Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study
JMIR Medical Informatics · 2026
DOI: 10.2196/73725 ↗At a glance
Validierungsstudie in JMIR Medical Informatics, 2026. Written by Kyumin Park and 15 co-authors. Institution: SoftlyAI, Seoul, Republic of Korea. Major MEDLINE topics: Suicide, Benchmarking. Cited 1 times so far (Crossref); 55 references. The full text is freely available in PubMed Central.
Abstract (original language)
Background
Harmful suicide content on the internet poses significant risks, as it can induce suicidal thoughts and behaviors, particularly among vulnerable populations. Despite global efforts, existing moderation approaches remain insufficient, especially in high-risk regions such as South Korea, which has the highest suicide rate among Organisation for Economic Co-operation and Development countries. Previous research has primarily focused on assessing the suicide risk of the authors who wrote the content rather than the harmfulness of content itself which potentially leads the readers to self-harm or suicide, highlighting a critical gap in current approaches. Our study addresses this gap by shifting the focus from assessing the suicide risk of content authors to evaluating the harmfulness of the content itself and its potential to induce suicide risk among readers.
Objective
This study aimed to develop an artificial intelligence (AI)–driven system for classifying online suicide-related content into 5 levels: illegal, harmful, potentially harmful, harmless, and non–suicide-related. In addition, the researchers construct a multimodal benchmark dataset with expert annotations to improve content moderation and assist AI models in detecting and regulating harmful content more effectively.
Methods
We collected 43,244 user-generated posts from various online sources, including social media, question and answer (Q&A) platforms, and online communities. To reduce the workload on human annotators, GPT-4 was used for preannotation, filtering, and categorizing content before manual review by medical professionals. A task description document ensured consistency in classification. Ultimately, a benchmark dataset of 452 manually labeled entries was developed, including both Korean and English versions, to support AI-based moderation. The study also evaluated zero-shot and few-shot learning to determine the best AI approach for detecting harmful content.
Results
The multimodal benchmark dataset showed that GPT-4 achieved the highest F 1 -scores (66.46 for illegal and 77.09 for harmful content detection). Image descriptions improved classification accuracy, while directly using raw images slightly decreased performance. Few-shot learning significantly enhanced detection, demonstrating that small but high-quality datasets could improve AI-driven moderation. However, translation challenges were observed, particularly in suicide-related slang and abbreviations, which were sometimes inaccurately conveyed in the English benchmark.
Conclusions
This study provides a high-quality benchmark for AI-based suicide content detection, proving that large language models can effectively assist in content moderation while reducing the burden on human moderators. Future work will focus on enhancing real-time detection and improving the handling of subtle or disguised harmful content.
Subjects
MeSH terms (MEDLINE)
Author keywords
Authors and institutions
- SoftlyAI, Seoul, Republic of Korea.
- Department of Psychiatry, Kyung Hee University College of Medicine, Seoul, Republic of Korea.
- Department of Artificial Intelligence, Sungkyunkwan University, Office 27306, Engineering Building 2, 2066 Seobu-ro Jangan-gu, Suwon-si, Gyeonggi-do, 16419, Republic of Korea, +82 31 290 7104.
- KAIST, Daejeon, Republic of Korea.
- Department of Psychology, University of Pennsylvania, Philadelphia, PA, United States.
- Department of Computer Science, New York University, New York, NY, United States.
ORCID profiles: Kyumin Park, Myung Jae Baik, YeongJun Hwang, Yen Shin, HoJae Lee, Ruda Lee, Sang Min Lee, Je Young Hannah Sun, Ah Rah Lee, Si Yeun Yoon, Dong-Ho Lee, Jihyung Moon, JinYeong Bak, Kyunghyun Cho, Jong-Woo Paik, Sungjoon Park
Cite
APA
Park, K., Baik, M. J., Hwang, Y., Shin, Y., Lee, H., Lee, R., Lee, S. M., Sun, J. Y. H., Lee, A. R., Yoon, S. Y., Lee, D. H., Moon, J., Bak, J., Cho, K., Paik, J. W., & Park, S. (2026). Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/73725
Vancouver
Park K, Baik MJ, Hwang Y, Shin Y, Lee H, Lee R, et al. Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study. JMIR Medical Informatics. 2026. doi:10.2196/73725
BibTeX
@article{Park2026,
author = {Park, Kyumin and Baik, Myung Jae and Hwang, YeongJun and Shin, Yen and Lee, HoJae and Lee, Ruda and Lee, Sang Min and Sun, Je Young Hannah and Lee, Ah Rah and Yoon, Si Yeun and Lee, Dong-Ho and Moon, Jihyung and Bak, JinYeong and Cho, Kyunghyun and Paik, Jong-Woo and Park, Sungjoon},
title = {Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study},
journal = {JMIR Medical Informatics},
year = {2026},
doi = {10.2196/73725}
}
EUVIMED