Efficient language identification in monolingual and multilingual documents
Journal of High School Science · 02.04.2026 · 10 (2)
DOI: 10.64336/001c.160011 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of High School Science, online erschienen am 02.04.2026. Verfasst von Maxime Sobrier. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-SA 4.0).
Abstract (Originalsprache)
Language identification (LID) is a critical prerequisite for text processing tasks such as content classification, natural language processing (NLP), machine translation, and large language model (LLM) training. Modern AI systems rely heavily on web data, making accurate detection essential for applying language-specific preprocessing techniques and aligning multilingual datasets. Existing LID systems, however, struggle with multilingual documents, short or noisy text, and resource constraints. This study introduces a lightweight term-frequency-based algorithm for detecting multiple languages in a single document, achieving high accuracy while minimizing memory usage. The algorithm assigns independent relevance scores to each language (not calibrated as proportions), enabling effective identification of both monolingual and multilingual content. The proposed method was benchmarked against state-of-the-art libraries, including BERT, CLD3, fastText, and GlotLID, on WiLI-2018, FLORES+, Tatoeba, OpenSubtitles, and a newly constructed Multilingual Webpages dataset. Results show that the algorithm achieves an F1 score of up to 98%, matching or exceeding existing libraries under comparable evaluation settings while remaining computationally efficient on standard hardware. Its ability to detect dominant languages in multilingual webpages further demonstrates its practical applicability.
Zitieren
APA
Sobrier, M. (2026). Efficient language identification in monolingual and multilingual documents. Journal of High School Science, 10(2). https://doi.org/10.64336/001c.160011
Vancouver
Sobrier M. Efficient language identification in monolingual and multilingual documents. Journal of High School Science. 2026;10(2). doi:10.64336/001c.160011
BibTeX
@article{Sobrier2026,
author = {Sobrier, Maxime},
title = {Efficient language identification in monolingual and multilingual documents},
journal = {Journal of High School Science},
year = {2026},
volume = {10},
number = {2},
doi = {10.64336/001c.160011}
}
EUVIMED