NormasTCU — A Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment
Information Processing & Management · 2027-02 · 64 (2) · 105151
DOI: 10.1016/j.ipm.2026.105151 ↗Kurz gefasst
Fachartikel in der Zeitschrift Information Processing & Management, 2027-02. Verfasst von Leandro Carísio Fernandes und 4 weiteren Autor:innen. Die Arbeit nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Coordination for the improvement of Higher Education Personnel.
Abstract (Originalsprache)
Portuguese Information Retrieval (IR) lacks public datasets, and relevance assessment for specialized collections remains costly. While Large Language Models (LLMs) increasingly support relevance assessment, their reliability in non-English specialized domains remains unclear. We introduce NormasTCU (https://huggingface.co/datasets/LeandroRibeiro/NormasTCU), a Brazilian Portuguese IR dataset with 14,469 legal documents, 46 queries, and 3,048 human judgments over 812 query-document pairs. Using NormasTCU, we evaluated LLM-as-a-judge for relevance assessment by prompting three models with two prompt techniques to grade these pairs. We then compared the rankings of 15 IR systems derived from LLM-generated and human reference qrels. LLMs consistently showed a positive scoring bias (mean absolute error: 0.46--0.66 on a 0-2 scale). Furthermore, pair-level agreement with human judgments achieved only fair to moderate levels, with Cohen's kappa ranging from 0.32 to 0.53. Despite this bias, LLM-generated judgments often yielded highly similar system rankings for nDCG@10 and MRR (observed Kendall's tau greater than or equal 0.90, although the bootstrap confidence intervals did not always remain above this threshold), but were less reliable for P@10 and R@10. Notably, LLM-based rankings were sometimes more strongly correlated with the reference ranking than individual human annotations were. As a practical implication, our results suggest that LLMs could effectively support scalable relevance assessment in specialized Portuguese corpora when evaluated using nDCG or MRR (rank-aware metrics), but they should be avoided when relying on precision or recall.
Förderung
- Coordination for the improvement of Higher Education Personnel
Zitieren
APA
Fernandes, L. C., Castro, M. V. B. D., Ribeiro, L. D. S., Pacheco, L. A. D. S., & Sandes, E. F. D. O. (2027). NormasTCU — A Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment. Information Processing & Management, 64(2), 105151. https://doi.org/10.1016/j.ipm.2026.105151
Vancouver
Fernandes LC, Castro MVBD, Ribeiro LDS, Pacheco LADS, Sandes EFDO. NormasTCU — A Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment. Information Processing & Management. 2027;64(2):105151. doi:10.1016/j.ipm.2026.105151
BibTeX
@article{Fernandes2027,
author = {Fernandes, Leandro Carísio and Castro, Marcus Vinícius Borela de and Ribeiro, Leandro dos Santos and Pacheco, Leonardo Augusto da Silva and Sandes, Edans Flávius de Oliveira},
title = {NormasTCU — A Brazilian Portuguese IR dataset and an evaluation of LLM-as-a-judge for relevance assessment},
journal = {Information Processing & Management},
year = {2027},
volume = {64},
number = {2},
pages = {105151},
doi = {10.1016/j.ipm.2026.105151}
}
EUVIMED