EUVIMEDOpdag EUVIMED
ValidierungsstudieOpen access

Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study

Kyumin Park iD, Myung Jae Baik iD, YeongJun Hwang iD, Yen Shin iD, HoJae Lee iD, Ruda Lee iD, Sang Min Lee iD, Je Young Hannah Sun iD, Ah Rah Lee iD, Si Yeun Yoon iD, Dong-Ho Lee iD, Jihyung Moon iD and 4 more

JMIR Medical Informatics · 2026

DOI: 10.2196/73725 ↗

At a glance

Validierungsstudie in JMIR Medical Informatics, 2026. Written by Kyumin Park and 15 co-authors. Institution: SoftlyAI, Seoul, Republic of Korea. Major MEDLINE topics: Suicide, Benchmarking. Cited 1 times so far (Crossref); 55 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Harmful suicide content on the internet poses significant risks, as it can induce suicidal thoughts and behaviors, particularly among vulnerable populations. Despite global efforts, existing moderation approaches remain insufficient, especially in high-risk regions such as South Korea, which has the highest suicide rate among Organisation for Economic Co-operation and Development countries. Previous research has primarily focused on assessing the suicide risk of the authors who wrote the content rather than the harmfulness of content itself which potentially leads the readers to self-harm or suicide, highlighting a critical gap in current approaches. Our study addresses this gap by shifting the focus from assessing the suicide risk of content authors to evaluating the harmfulness of the content itself and its potential to induce suicide risk among readers.

Objective

This study aimed to develop an artificial intelligence (AI)–driven system for classifying online suicide-related content into 5 levels: illegal, harmful, potentially harmful, harmless, and non–suicide-related. In addition, the researchers construct a multimodal benchmark dataset with expert annotations to improve content moderation and assist AI models in detecting and regulating harmful content more effectively.

Methods

We collected 43,244 user-generated posts from various online sources, including social media, question and answer (Q&A) platforms, and online communities. To reduce the workload on human annotators, GPT-4 was used for preannotation, filtering, and categorizing content before manual review by medical professionals. A task description document ensured consistency in classification. Ultimately, a benchmark dataset of 452 manually labeled entries was developed, including both Korean and English versions, to support AI-based moderation. The study also evaluated zero-shot and few-shot learning to determine the best AI approach for detecting harmful content.

Results

The multimodal benchmark dataset showed that GPT-4 achieved the highest F 1 -scores (66.46 for illegal and 77.09 for harmful content detection). Image descriptions improved classification accuracy, while directly using raw images slightly decreased performance. Few-shot learning significantly enhanced detection, demonstrating that small but high-quality datasets could improve AI-driven moderation. However, translation challenges were observed, particularly in suicide-related slang and abbreviations, which were sometimes inaccurately conveyed in the English benchmark.

Conclusions

This study provides a high-quality benchmark for AI-based suicide content detection, proving that large language models can effectively assist in content moderation while reducing the burden on human moderators. Future work will focus on enhancing real-time detection and improving the handling of subtle or disguised harmful content.

Subjects

MeSH terms (MEDLINE)

SuicideBenchmarkingLarge Language ModelsHumansRepublic of Korea

Author keywords

artificial intelligencedatasetlarge language modelssuicidesuicide-related content

Authors and institutions

  • SoftlyAI, Seoul, Republic of Korea.
  • Department of Psychiatry, Kyung Hee University College of Medicine, Seoul, Republic of Korea.
  • Department of Artificial Intelligence, Sungkyunkwan University, Office 27306, Engineering Building 2, 2066 Seobu-ro Jangan-gu, Suwon-si, Gyeonggi-do, 16419, Republic of Korea, +82 31 290 7104.
  • KAIST, Daejeon, Republic of Korea.
  • Department of Psychology, University of Pennsylvania, Philadelphia, PA, United States.
  • Department of Computer Science, New York University, New York, NY, United States.

ORCID profiles: Kyumin Park, Myung Jae Baik, YeongJun Hwang, Yen Shin, HoJae Lee, Ruda Lee, Sang Min Lee, Je Young Hannah Sun, Ah Rah Lee, Si Yeun Yoon, Dong-Ho Lee, Jihyung Moon, JinYeong Bak, Kyunghyun Cho, Jong-Woo Paik, Sungjoon Park

Cite

APA

Park, K., Baik, M. J., Hwang, Y., Shin, Y., Lee, H., Lee, R., Lee, S. M., Sun, J. Y. H., Lee, A. R., Yoon, S. Y., Lee, D. H., Moon, J., Bak, J., Cho, K., Paik, J. W., & Park, S. (2026). Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study. JMIR Medical Informatics. https://doi.org/10.2196/73725

Vancouver

Park K, Baik MJ, Hwang Y, Shin Y, Lee H, Lee R, et al. Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study. JMIR Medical Informatics. 2026. doi:10.2196/73725

BibTeX
@article{Park2026,
  author = {Park, Kyumin and Baik, Myung Jae and Hwang, YeongJun and Shin, Yen and Lee, HoJae and Lee, Ruda and Lee, Sang Min and Sun, Je Young Hannah and Lee, Ah Rah and Yoon, Si Yeun and Lee, Dong-Ho and Moon, Jihyung and Bak, JinYeong and Cho, Kyunghyun and Paik, Jong-Woo and Park, Sungjoon},
  title = {Iterative Large Language Model–Guided Sampling and Expert-Annotated Benchmark Corpus for Harmful Suicide Content Detection: Development and Validation Study},
  journal = {JMIR Medical Informatics},
  year = {2026},
  doi = {10.2196/73725}
}
Start for free →