Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment
Electronics · 2026-11-19 · 15 (22) · 5394
DOI: 10.3390/electronics15225394 ↗At a glance
Journal article in Electronics, published online on 2026-11-19. Written by Yunam Cheong and 2 co-authors. 30 references. The full text is freely available (open access, CC BY 4.0). Funded by: Sunchon National University Glocal University Project Fund.
Abstract (original language)
Toxic language detection for large language model safety has largely been framed as binary classification, yet unethical expressions differ in severity across types and human judgments of severity are far from uniform. We analyze a large-scale Korean ethics-verification corpus with eight type labels (seven unethical types plus neutral) and the individual intensity votes (0–3) of five raters per sentence (363,154 sentences; 1,001,030 votes). First, the seven unethical types form a three-tier perceived-severity hierarchy—censure (1.42) < discrimination, hate, and abuse (1.53–1.59) < sexual, violence and crime incitement (1.74–1.75)—supported by post-hoc tests: all between-tier differences are significant with substantial effect sizes (Holm-corrected p < 0.05; |Cliff’s δ| = 0.14–0.45), while within-tier effects are negligible. Second, automatically estimated sentiment valence barely explains the hierarchy (Spearman ρ = −0.142; AUC = 0.621; replicated with a Korean-specific sentiment model and with preceding dialogue context included in the input); the two most severe types show the most neutral surface valence, so sentiment signals capture best precisely the least severe type. Third, inter-rater agreement is low (ordinal Krippendorff’s α = 0.373), yet the disagreement is not explained by the basic demographics the corpus records—rater gender and age (equivalence tests bound the pooled gender effect at |d|≤0.013 and the age correlation at |ρ|≤0.002; unrecorded rater characteristics remain untested)—and instead grows with item severity (sentence-level ρ = 0.248). Harmfulness judgment appears to be a dimension largely independent of sentiment, whose annotation may benefit from continuous scales, multi-rater averaging, and severity-proportional rating resources.
Funding
- Sunchon National University Glocal University Project Fund
Cite
APA
Cheong, Y., Shin, S., & Shin, K. (2026). Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment. Electronics, 15(22), 5394. https://doi.org/10.3390/electronics15225394
Vancouver
Cheong Y, Shin S, Shin K. Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment. Electronics. 2026;15(22):5394. doi:10.3390/electronics15225394
BibTeX
@article{Cheong2026,
author = {Cheong, Yunam and Shin, Seong-Yoon and Shin, Kwang-Seong},
title = {Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment},
journal = {Electronics},
year = {2026},
volume = {15},
number = {22},
pages = {5394},
doi = {10.3390/electronics15225394}
}
EUVIMED