EUVIMEDEUVIMED entdecken
Crossref

naab: A ready-to-use plug-and-play corpus for Farsi

Sadra Sabouri, Elnaz Rahmati Rahmati, Soroush Gooran, Hossein Sameti

Journal of Artificial Intelligence, Applications, and Innovations · 2024

DOI: 10.61838/jaiai.1.2.1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Artificial Intelligence, Applications, and Innovations, 2024. Verfasst von Sadra Sabouri und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The rise of large language models (LLMs) has transformed numerous natural language processing (NLP) tasks, yet their performance in low and mid-resource languages, such as Farsi, still lags behind resource-rich languages like English. To address this gap, we introduce Naab, the largest publicly available, cleaned, and ready-to-use Farsi textual corpus. Naab consists of 130GB of data, comprising over 250 million paragraphs and 15 billion words. Named after the Farsi word ناب (meaning "pure" or "high-grade"), this corpus is openly accessible via Hugging Face, offering researchers a valuable resource for Farsi NLP tasks. In addition to naab, we provide naab-raw, an unprocessed version of the dataset, along with a pre-processing toolkit that allows users to clean their custom corpora. These resources empower NLP researchers and practitioners, particularly those focusing on low-resource languages, to improve the performance of LLMs in their respective domains and bridge the gap between resource-rich and resource-poor languages.

Zitieren

APA

Sabouri, S., Rahmati, E. R., Gooran, S., & Sameti, H. (2024). naab: A ready-to-use plug-and-play corpus for Farsi. Journal of Artificial Intelligence, Applications, and Innovations. https://doi.org/10.61838/jaiai.1.2.1

Vancouver

Sabouri S, Rahmati ER, Gooran S, Sameti H. naab: A ready-to-use plug-and-play corpus for Farsi. Journal of Artificial Intelligence, Applications, and Innovations. 2024. doi:10.61838/jaiai.1.2.1

BibTeX
@article{Sabouri2024,
  author = {Sabouri, Sadra and Rahmati, Elnaz Rahmati and Gooran, Soroush and Sameti, Hossein},
  title = {naab: A ready-to-use plug-and-play corpus for Farsi},
  journal = {Journal of Artificial Intelligence, Applications, and Innovations},
  year = {2024},
  doi = {10.61838/jaiai.1.2.1}
}
Kostenlos starten →