EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Annotation Reliability and Classification Performance on Hate Speech Detection in YouTube Discussions on VAT Increases

Lathifah Alfat, Raden Rara Kartika Kusuma Winahyu, Masurah Mohamad

Jurnal Informatika: Jurnal Pengembangan IT · 04.11.2026 · 11 (3) · 996-1006

DOI: 10.30591/jpit.v11i3.130 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Jurnal Informatika: Jurnal Pengembangan IT, online erschienen am 04.11.2026. Verfasst von Lathifah Alfat und 2 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

use of social media as a medium for complaints and conveying public opinion has increased over the years. However, this situation ignites hate speech and harsh words that potentially disturb the digital discussion space. The purpose of this research is to analyze the influence of labelling schemes on annotation and classification performance of hate speech models in YouTube comments, related to Value Added Tax increases in Indonesia. The data consist of 6,397 comments that were annotated using two labelling schemes: three classes (normal, abusive, and hate) and two classes (toxic and non-toxic). The annotation process included two annotators, one rule-based and one human, and the result was evaluated using Cohen’s Kappa. The evaluation shows that the two-class scheme results in better annotation reliability, with Cohen’s Kappa of 0.641 (substantial agreement) versus 0.552 in the three-class scheme. The experiment continued with classification using IndoBERT, SVM based on TF-IDF, and CatBoost. In the three classes, IndoBERT got the highest Macro-F1 value, about 0.3778, increasing to 0.4126 after the implementation of class weight to overcome the class imbalance. In the two-class scheme, IndoBERT got a Macro-F1 of 0.6084, but accuracy (0.8033) was below the majority baseline (0.827). The research shows that joining the “hate” and “abusive” labels boosts the consistency of annotation but does not reliably improve the classification model performance. This finding indicates that label granularity affects annotation quality, but its effect on hate speech model performance in public policy in the Indonesian language is inconclusive.

Zitieren

APA

Alfat, L., Winahyu, R. R. K. K., & Mohamad, M. (2026). Annotation Reliability and Classification Performance on Hate Speech Detection in YouTube Discussions on VAT Increases. Jurnal Informatika: Jurnal Pengembangan IT, 11(3), 996-1006. https://doi.org/10.30591/jpit.v11i3.130

Vancouver

Alfat L, Winahyu RRKK, Mohamad M. Annotation Reliability and Classification Performance on Hate Speech Detection in YouTube Discussions on VAT Increases. Jurnal Informatika: Jurnal Pengembangan IT. 2026;11(3):996-1006. doi:10.30591/jpit.v11i3.130

BibTeX
@article{Alfat2026,
  author = {Alfat, Lathifah and Winahyu, Raden Rara Kartika Kusuma and Mohamad, Masurah},
  title = {Annotation Reliability and Classification Performance on Hate Speech Detection in YouTube Discussions on VAT Increases},
  journal = {Jurnal Informatika: Jurnal Pengembangan IT},
  year = {2026},
  volume = {11},
  number = {3},
  pages = {996--1006},
  doi = {10.30591/jpit.v11i3.130}
}
Kostenlos starten →