EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

Beyond Flip Rate: A Multi-Criteria Comparative Benchmark of Counterfactual Explanation Methods for Indonesian Text Classification

Farida Yusuf, Firmansyah Ibrahim

Information Technology Education Journal · 01.11.2026 · 33-55

DOI: 10.59562/intec.v5i4.13987 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Information Technology Education Journal, online erschienen am 01.11.2026. Verfasst von Farida Yusuf und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose – A counterfactual explanation for Indonesian text classification must simultaneously flip the classifier’s decision, remain a minimal and fluent edit, and preserve content that has no bearing on the label. This study asks whether a more powerful generator yields a better Indonesian counterfactual and benchmarks five generators under a single multi-criteria framework. Design/methods/approach – Five generators, a masked-language substitution baseline, two zero-shot LLM pipelines, a gradient-guided variant, and a second-family LLM, were evaluated on two Indonesian test sets across validity, edit quality, format compliance, and cost, combined into a single composite index, and tested for statistical significance. Findings: The masked-language baseline is the strongest and cheapest generator, leading on both flip rate and composite quality while running approximately 55 to 198 times faster than the LLM pipelines. The LLM pipelines are competitive on edit quality but not on validity, and the second-family model degenerates on roughly two-thirds of its outputs, a failure that a flip-rate-only evaluation would have masked. Research implications/limitations: For practitioners building trustworthy AI systems, the practical implication is that a cheaper, simpler generator can be a safer choice, and that generator selection should rest on multi-axis, cost-aware evidence rather than on flip rate or model size alone. All axes here are automated proxies evaluated on two tasks without human judgement. Originality/value – To the best of our knowledge, this study presents the first multicriteria counterfactual benchmark for Indonesian text classification, showing that generator size is a poor proxy for explanation quality and offering a reusable, cost-aware protocol for selecting trustworthy counterfactual methods in lower-resourced languages.

Zitieren

APA

Yusuf, F., & Ibrahim, F. (2026). Beyond Flip Rate: A Multi-Criteria Comparative Benchmark of Counterfactual Explanation Methods for Indonesian Text Classification. Information Technology Education Journal, 33-55. https://doi.org/10.59562/intec.v5i4.13987

Vancouver

Yusuf F, Ibrahim F. Beyond Flip Rate: A Multi-Criteria Comparative Benchmark of Counterfactual Explanation Methods for Indonesian Text Classification. Information Technology Education Journal. 2026:33-55. doi:10.59562/intec.v5i4.13987

BibTeX
@article{Yusuf2026,
  author = {Yusuf, Farida and Ibrahim, Firmansyah},
  title = {Beyond Flip Rate: A Multi-Criteria Comparative Benchmark of Counterfactual Explanation Methods for Indonesian Text Classification},
  journal = {Information Technology Education Journal},
  year = {2026},
  pages = {33--55},
  doi = {10.59562/intec.v5i4.13987}
}
Kostenlos starten →