Towards Romanian Large Language Models Using the CoRoLa Corpus
Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing" · 31.12.2025 · 235
DOI: 10.47743/consilr2025.16 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing", online erschienen am 31.12.2025. Verfasst von Vasile Păiș und 7 weiteren Autor:innen.
Abstract (Originalsprache)
Recent developments of large language models (LLMs) focus on extensive volumes of crawled data, employing datasets such as Common Crawl. While this ensures a high number of tokens in many languages, it raises intellectual property rights issues. Furthermore, the quality of the crawled data is lower (from both a linguistic and a scientific perspective) compared to books or other peer-reviewed publications. The Representative Corpus of the Contemporary Romanian language (CoRoLa) is a high quality, IPR cleared, large language resource. However, it is orders of magnitude smaller compared to unconstrained crawled data. In this paper we explore the impact of including such a high-quality but smaller resource into LLMs by means of continuous pretraining, targeting the Romanian language. We report our experiments on a small Llama-based model and express our concerns regarding existing datasets for the Romanian language. We further propose ways to potentially improve Romanian language LLMs by creating new datasets of high quality, both textual and instruction-based.
Zitieren
APA
Păiș, V., Tufiș, D., Mititelu, V. B., Ion, R., Mitrofan, M., Irimia, E., Curea, E., & Păun, A. (2025). Towards Romanian Large Language Models Using the CoRoLa Corpus. Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing", 235. https://doi.org/10.47743/consilr2025.16
Vancouver
Păiș V, Tufiș D, Mititelu VB, Ion R, Mitrofan M, Irimia E, et al. Towards Romanian Large Language Models Using the CoRoLa Corpus. Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing". 2025:235. doi:10.47743/consilr2025.16
BibTeX
@article{Pi2025,
author = {Păiș, Vasile and Tufiș, Dan and Mititelu, Verginica Barbu and Ion, Radu and Mitrofan, Maria and Irimia, Elena and Curea, Eric and Păun, Andrei},
title = {Towards Romanian Large Language Models Using the CoRoLa Corpus},
journal = {Proceedings of the ... International Conference "Linguistic Resources and Tools for Natural Language Processing"},
year = {2025},
pages = {235},
doi = {10.47743/consilr2025.16}
}
EUVIMED