Exploring Brazil's LLM Fauna: Investigating the Generative Performance of Large Language Models in Portuguese
Journal of the Brazilian Computer Society · 08.10.2025 · 31 (1) · 940-972
DOI: 10.5753/jbcs.2025.5814 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of the Brazilian Computer Society, online erschienen am 08.10.2025. Verfasst von Gabriel Assis und 2 weiteren Autor:innen. Die Arbeit wurde bisher 5-mal zitiert (Crossref) und nennt 171 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Large Language Models (LLMs) are now embedded in widely used applications worldwide, yet their evaluation still centers on narrow, discriminative benchmarks. These pipelines often overlook key generative aspects such as discourse coherence, linguistic transformations, and adequacy, which are crucial for real-world applications. In addition, most large-scale evaluations remain heavily biased toward English, limiting our understanding of LLM performance in other languages. This research addresses these gaps by presenting a comprehensive analysis of Brazilian Portuguese LLMs across three core Natural Language Generation tasks: summarization, simplification, and generative question answering. We evaluate six Brazilian models and compare them to the widely used GPT-4o. Our findings, supported by diverse automatic metrics, an LLM-as-a-judge framework, and human evaluation, show that GPT-4o series achieves the best generative performance in Portuguese, followed closely by the Sabiá-3 family. While slightly behind, the open-weight model Tucano stands out for its computational efficiency, making it a strong candidate for deployment in resource-constrained settings. The code used to conduct all experiments is publicly available at https://github.com/MeLLL-UFF/brfauna-gen-eval.
Zitieren
APA
Assis, G., Freitas, C., & Paes, A. (2025). Exploring Brazil's LLM Fauna: Investigating the Generative Performance of Large Language Models in Portuguese. Journal of the Brazilian Computer Society, 31(1), 940-972. https://doi.org/10.5753/jbcs.2025.5814
Vancouver
Assis G, Freitas C, Paes A. Exploring Brazil's LLM Fauna: Investigating the Generative Performance of Large Language Models in Portuguese. Journal of the Brazilian Computer Society. 2025;31(1):940-972. doi:10.5753/jbcs.2025.5814
BibTeX
@article{Assis2025,
author = {Assis, Gabriel and Freitas, Cláudia and Paes, Aline},
title = {Exploring Brazil's LLM Fauna: Investigating the Generative Performance of Large Language Models in Portuguese},
journal = {Journal of the Brazilian Computer Society},
year = {2025},
volume = {31},
number = {1},
pages = {940--972},
doi = {10.5753/jbcs.2025.5814}
}
EUVIMED