AtlasBERT: a modernized encoder for Turkish general and clinical language inference
PeerJ Computer Science · 19.11.2026 · 12 · e4173
DOI: 10.7717/peerj-cs.4173 ↗Kurz gefasst
Fachartikel in der Zeitschrift PeerJ Computer Science, online erschienen am 19.11.2026. Verfasst von İskender Ülgen Oğul und 2 weiteren Autor:innen. Die Arbeit nennt 48 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Scientific and Technological Research Council of Türkiye (TÜBİTAK) through the BİDEB 2211-C Priority Areas Domestic Doctoral Scholarship Programme, Scientific Research Projects (BAP) Coordination Unit of Aydın Adnan Menderes University.
Abstract (Originalsprache)
Decoder-only language models have advanced rapidly through architectural innovations such as new normalization methods, gated activations, and efficient attention mechanisms. However, encoder-only architectures, which are excellent at capturing contextual information for tasks like sequence classification and Natural Language Inference (NLI), have largely retained the original BERT family’s design principles. This gap creates particular challenges for low-resource languages like Turkish, where the scarcity of domain-specific pretrained encoders limits progress in specialized areas such as clinical natural language processing (NLP). Although English biomedical and clinical encoders have demonstrated that domain-adaptive pretraining improves biomedical and clinical NLP performance, comparable resources for Turkish remain underdeveloped. To address this, we present Atlas bidirectional encoder representations from transformers (BERT), a modernized BERT-style encoder that integrates recent innovations from both encoder and decoder models for Turkish general and clinical language inference. Inspired by models like ModernBERT, NeoBERT, and EuroBERT, our architecture employs pre-RMSNorm, SwiGLU feed-forward layers, and PyTorch FlexAttention with document-aware masking to extend the context window to 2,048 tokens while preserving document boundaries. Additionally, to address Turkish’s complex agglutinative morphology, we employ a SentencePiece Unigram tokenizer for subword segmentation. AtlasBERT is trained in two stages, first on 35.68 billion Turkish tokens from FineWeb-2, then with continued domain-adaptive pretraining on 791 million tokens of Turkish biomedical text. AtlasBERT achieves 87.82% mean test accuracy on Turkish Stanford natural language inference (SNLI), compared with BERTurk’s 87.03%, while in the clinical domain BioAtlasBERT reaches 77.80% on TurkMedNLI against a BioBERTurk baseline of 75.28%. These results show that combining modern encoder components with Turkish-tailored tokenization and two-stage pretraining provides a competitive foundation for Turkish general and clinical NLI.
Förderung
- Scientific and Technological Research Council of Türkiye (TÜBİTAK) through the BİDEB 2211-C Priority Areas Domestic Doctoral Scholarship Programme
- Scientific Research Projects (BAP) Coordination Unit of Aydın Adnan Menderes University
Zitieren
APA
Oğul, İ. Ü., Soygazi, F., & Bostanoğlu, B. E. (2026). AtlasBERT: a modernized encoder for Turkish general and clinical language inference. PeerJ Computer Science, 12, e4173. https://doi.org/10.7717/peerj-cs.4173
Vancouver
Oğul İÜ, Soygazi F, Bostanoğlu BE. AtlasBERT: a modernized encoder for Turkish general and clinical language inference. PeerJ Computer Science. 2026;12:e4173. doi:10.7717/peerj-cs.4173
BibTeX
@article{Oul2026,
author = {Oğul, İskender Ülgen and Soygazi, Fatih and Bostanoğlu, Belgin Ergenç},
title = {AtlasBERT: a modernized encoder for Turkish general and clinical language inference},
journal = {PeerJ Computer Science},
year = {2026},
volume = {12},
pages = {e4173},
doi = {10.7717/peerj-cs.4173}
}
EUVIMED