EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Benchmarking Prompt Injection Attacks on LLMs: Turkish Vulnerability Assessment and English Comparative Analysis

Öner Aytaş, Tuğçe Şen, Banu Diri, Göksel Biricik, Mehmet Ali Bayram

Applied Sciences · 06.07.2026 · 16 (13) · 6740

DOI: 10.3390/app16136740 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Applied Sciences, online erschienen am 06.07.2026. Verfasst von Öner Aytaş und 4 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 39 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Scientific and Technological Research Council of Türkiye.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) are increasingly deployed in multilingual settings, yet their safety behavior under Turkish harmful prompts and prompt injection attempts remains insufficiently characterized. This study evaluates the adversarial robustness of 55 open- and closed-source LLMs under paired Turkish and English harmful prompt conditions. We constructed a benchmark of 790 Turkish adversarial prompts, translated the prompts into English for cross-lingual comparison, and applied both prompt sets to the model pool. Model responses were labeled as harmful, harmless, or hallucinatory, and safety was analyzed using safety scores, Turkish–English ranking differences, and inter-rater reliability based on Fleiss’ kappa. The results reveal substantial variation across models. Closed-source systems generally achieved higher safety scores and stronger filtering behavior, whereas open-source and Turkish-oriented models showed a wider performance distribution. GPT-5.4 ranked first in the Turkish tests with a 99.37% safety score but decreased to 96.71% in the English tests, while Qwen3.5:27B ranked first in English with 97.47%. These differences suggest that safety mechanisms are not fully language-invariant. Hallucination also emerged as a distinct safety risk, particularly in Turkish evaluations. The findings indicate that Turkish LLM safety cannot be inferred from general model capability alone and should be assessed through language-specific, culturally aware, and continuously updated adversarial benchmarks.

Förderung

  • Scientific and Technological Research Council of Türkiye

Zitieren

APA

Aytaş, Ö., Şen, T., Diri, B., Biricik, G., & Bayram, M. A. (2026). Benchmarking Prompt Injection Attacks on LLMs: Turkish Vulnerability Assessment and English Comparative Analysis. Applied Sciences, 16(13), 6740. https://doi.org/10.3390/app16136740

Vancouver

Aytaş Ö, Şen T, Diri B, Biricik G, Bayram MA. Benchmarking Prompt Injection Attacks on LLMs: Turkish Vulnerability Assessment and English Comparative Analysis. Applied Sciences. 2026;16(13):6740. doi:10.3390/app16136740

BibTeX
@article{Ayta2026,
  author = {Aytaş, Öner and Şen, Tuğçe and Diri, Banu and Biricik, Göksel and Bayram, Mehmet Ali},
  title = {Benchmarking Prompt Injection Attacks on LLMs: Turkish Vulnerability Assessment and English Comparative Analysis},
  journal = {Applied Sciences},
  year = {2026},
  volume = {16},
  number = {13},
  pages = {6740},
  doi = {10.3390/app16136740}
}
Kostenlos starten →