EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Toward purpose-oriented topic model evaluation enabled by large language models

Zhiyin Tan, Jennifer D’Souza

International Journal on Digital Libraries · 2025-12 · 26 (4)

DOI: 10.1007/s00799-025-00429-5 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal on Digital Libraries, online erschienen am 22.10.2025. Verfasst von Zhiyin Tan und 1 weiteren Autor:innen. Die Arbeit wurde bisher 12-mal zitiert (Crossref) und nennt 64 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Niedersächsisches Ministerium für Wissenschaft und Kultur, Bundesministerium für Bildung und Forschung.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

This study presents a framework for automated evaluation of dynamically evolving topic models using Large Language Models (LLMs). Topic modeling is essential for organizing and retrieving scholarly content in digital library systems, helping users navigate complex and evolving knowledge domains. However, widely used automated metrics, such as coherence and diversity, often capture only narrow statistical patterns and fail to explain semantic failures in practice. We introduce a purpose-oriented evaluation framework that employs nine LLM-based metrics spanning four key dimensions of topic quality: lexical validity, intra-topic semantic soundness, inter-topic structural soundness, and document-topic alignment soundness. The framework is validated through adversarial and sampling-based protocols, and is applied across datasets spanning news articles, scholarly publications, and social media posts, as well as multiple topic modeling methods and open-source LLMs. Our analysis shows that LLM-based metrics provide interpretable, robust, and task-relevant assessments, uncovering critical weaknesses in topic models such as redundancy and semantic drift, which are often missed by traditional metrics. These results support the development of scalable, fine-grained evaluation tools for maintaining topic relevance in dynamic datasets.

Förderung

  • Niedersächsisches Ministerium für Wissenschaft und Kultur
  • Bundesministerium für Bildung und Forschung

Zitieren

APA

Tan, Z., & D’Souza, J. (2025). Toward purpose-oriented topic model evaluation enabled by large language models. International Journal on Digital Libraries, 26(4). https://doi.org/10.1007/s00799-025-00429-5

Vancouver

Tan Z, D’Souza J. Toward purpose-oriented topic model evaluation enabled by large language models. International Journal on Digital Libraries. 2025;26(4). doi:10.1007/s00799-025-00429-5

BibTeX
@article{Tan2025,
  author = {Tan, Zhiyin and D’Souza, Jennifer},
  title = {Toward purpose-oriented topic model evaluation enabled by large language models},
  journal = {International Journal on Digital Libraries},
  year = {2025},
  volume = {26},
  number = {4},
  doi = {10.1007/s00799-025-00429-5}
}
Kostenlos starten →