EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

SACHN-DeBERTa-v3-Large: Automated Document Security Classification with XAI and LLM Comparison

Mehmet Tuğrul Sariçiçek, Murat Dener

Applied Sciences · 04.06.2026 · 16 (11) · 5661

DOI: 10.3390/app16115661 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 04.06.2026. Verfasst von Mehmet Tuğrul Sariçiçek und 1 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 75 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Gazi University.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The manual classification of diplomatic documents by security sensitivity level is labor-intensive and inconsistent at scale. This study proposes SACHN-DeBERTa-v3-Large, a hybrid classification architecture integrating a DeBERTa-v3-Large backbone with a Security-Aware Gate, Prototype Classification Layer, and Supervised Contrastive Projection Head. A rule-based preprocessing pipeline removes inline classification markers prior to training, ensuring that models learn semantic content rather than surface-form annotation artifacts. Two diplomatic corpora are evaluated: the WikiLeaks Cable Classifier (9005 documents, three classes) and a novel Foreign Relations of the United States (FRUS) dataset (24,706 documents, four classes) constructed for this study. A controlled multi-LLM comparative study evaluates LLaMA-3-8B-Instruct, Qwen2.5-14B-Instruct, and Mistral-7B-Instruct-v0.3 under identical QLoRA fine-tuning conditions. SACHN-DeBERTa-v3-Large achieves 96.12% accuracy and 92.57% macro F1 on WikiLeaks, and 92.11% accuracy and 91.02% macro F1 on FRUS, surpassing all baselines by at least 10 percentage points in accuracy and 26 points in macro F1 (McNemar’s test, p < 0.001). Under the primary QLoRA protocol (r = 16), no evaluated LLM exceeded 57.16% accuracy; extended experiments with r = 64 and 8-bit INT8 quantisation yielded a best result of 64.48% (Mistral-7B-Instruct, WikiLeaks), confirming that the performance gap relative to SACHN-DeBERTa-v3-Large remains structural. Post-hoc SHAP and LIME analyses confirm that predictions are grounded in domain-specific semantic content, validating the classification marker removal methodology.

Förderung

  • Gazi University

Zitieren

APA

Sariçiçek, M. T., & Dener, M. (2026). SACHN-DeBERTa-v3-Large: Automated Document Security Classification with XAI and LLM Comparison. Applied Sciences, 16(11), 5661. https://doi.org/10.3390/app16115661

Vancouver

Sariçiçek MT, Dener M. SACHN-DeBERTa-v3-Large: Automated Document Security Classification with XAI and LLM Comparison. Applied Sciences. 2026;16(11):5661. doi:10.3390/app16115661

BibTeX
@article{Sariiek2026,
  author = {Sariçiçek, Mehmet Tuğrul and Dener, Murat},
  title = {SACHN-DeBERTa-v3-Large: Automated Document Security Classification with XAI and LLM Comparison},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {11},
  pages = {5661},
  doi = {10.3390/app16115661}
}
Kostenlos starten →