EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment

Yunam Cheong, Seong-Yoon Shin, Kwang-Seong Shin

Electronics · 19.11.2026 · 15 (22) · 5394

DOI: 10.3390/electronics15225394 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Electronics, online erschienen am 19.11.2026. Verfasst von Yunam Cheong und 2 weiteren Autor:innen. Die Arbeit nennt 30 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Sunchon National University Glocal University Project Fund.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Toxic language detection for large language model safety has largely been framed as binary classification, yet unethical expressions differ in severity across types and human judgments of severity are far from uniform. We analyze a large-scale Korean ethics-verification corpus with eight type labels (seven unethical types plus neutral) and the individual intensity votes (0–3) of five raters per sentence (363,154 sentences; 1,001,030 votes). First, the seven unethical types form a three-tier perceived-severity hierarchy—censure (1.42) < discrimination, hate, and abuse (1.53–1.59) < sexual, violence and crime incitement (1.74–1.75)—supported by post-hoc tests: all between-tier differences are significant with substantial effect sizes (Holm-corrected p < 0.05; |Cliff’s δ| = 0.14–0.45), while within-tier effects are negligible. Second, automatically estimated sentiment valence barely explains the hierarchy (Spearman ρ = −0.142; AUC = 0.621; replicated with a Korean-specific sentiment model and with preceding dialogue context included in the input); the two most severe types show the most neutral surface valence, so sentiment signals capture best precisely the least severe type. Third, inter-rater agreement is low (ordinal Krippendorff’s α = 0.373), yet the disagreement is not explained by the basic demographics the corpus records—rater gender and age (equivalence tests bound the pooled gender effect at |d|≤0.013 and the age correlation at |ρ|≤0.002; unrecorded rater characteristics remain untested)—and instead grows with item severity (sentence-level ρ = 0.248). Harmfulness judgment appears to be a dimension largely independent of sentiment, whose annotation may benefit from continuous scales, multi-rater averaging, and severity-proportional rating resources.

Förderung

  • Sunchon National University Glocal University Project Fund

Zitieren

APA

Cheong, Y., Shin, S., & Shin, K. (2026). Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment. Electronics, 15(22), 5394. https://doi.org/10.3390/electronics15225394

Vancouver

Cheong Y, Shin S, Shin K. Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment. Electronics. 2026;15(22):5394. doi:10.3390/electronics15225394

BibTeX
@article{Cheong2026,
  author = {Cheong, Yunam and Shin, Seong-Yoon and Shin, Kwang-Seong},
  title = {Perceived-Severity Hierarchy and Rating Disagreement in Korean Unethical Utterances: Harmfulness Judgments Are Not Reducible to Sentiment},
  journal = {Electronics},
  year = {2026},
  volume = {15},
  number = {22},
  pages = {5394},
  doi = {10.3390/electronics15225394}
}
Kostenlos starten →