Do LLMs agree on multi-category journal classification? Interpreting systematic divergence in Scopus-based journal categorization
Journal of Librarianship and Information Science · 29.07.2026
DOI: 10.1177/09610006261468375 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Librarianship and Information Science, online erschienen am 29.07.2026. Verfasst von Eungi Kim und 2 weiteren Autor:innen. Die Arbeit nennt 35 Literaturangaben. Gefördert von: Bisa Research Grant of Keimyung University.
Abstract (Originalsprache)
This paper aims to examine how four large language models (LLMs), Claude 3 Haiku, GPT 4o Mini, Gemini Flash 2.0, and DeepSeek Chat, classify 273 Library and Information Science (LIS) journals into 85 Scopus subject categories based on journal scope statements and to determine what their disagreements reveal about disciplinary boundary setting. Perfect agreement across all four models occurred in only 2.6% of cases and was confined to journals with unambiguous disciplinary positioning. Although the models matched the Scopus-assigned LIS category at rates ranging from 72.9% to 81.7%, they differed substantially in their assignment of additional categories, with some models consistently producing broader category sets and others adopting more restrictive classification strategies. A linear mixed-effects model confirmed that these differences were systematic rather than random. The findings have two important implications. First, set-overlap metrics such as Jaccard similarity inherently favor parsimonious classifiers irrespective of substantive accuracy. Second, the marked decline in agreement from the Scopus-assigned LIS category to additional categories suggests that classification systems may benefit from distinguishing between high-confidence and low-confidence assignments. Overall, inter-model disagreement appears to reflect the inherent multiplicity of disciplinary boundary setting rather than correctable technical error.
Förderung
- Bisa Research Grant of Keimyung University
Zitieren
APA
Kim, E., Min, S., & Chiu, J. L. (2026). Do LLMs agree on multi-category journal classification? Interpreting systematic divergence in Scopus-based journal categorization. Journal of Librarianship and Information Science. https://doi.org/10.1177/09610006261468375
Vancouver
Kim E, Min S, Chiu JL. Do LLMs agree on multi-category journal classification? Interpreting systematic divergence in Scopus-based journal categorization. Journal of Librarianship and Information Science. 2026. doi:10.1177/09610006261468375
BibTeX
@article{Kim2026,
author = {Kim, Eungi and Min, Sein and Chiu, Jason Lim},
title = {Do LLMs agree on multi-category journal classification? Interpreting systematic divergence in Scopus-based journal categorization},
journal = {Journal of Librarianship and Information Science},
year = {2026},
doi = {10.1177/09610006261468375}
}
EUVIMED