EUVIMEDEUVIMED entdecken
Crossref

Are Large Language Models a Good Replacement of Taxonomies?

Yushi Sun, Hao Xin, Kai Sun, Yifan Ethan Xu, Xiao Yang, Xin Luna Dong, Nan Tang, Lei Chen

Proceedings of the VLDB Endowment · 2024-07 · 17 (11) · 2919-2932

DOI: 10.14778/3681954.3681973 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the VLDB Endowment, online erschienen am 30.08.2024. Verfasst von Yushi Sun und 7 weiteren Autor:innen. Die Arbeit wurde bisher 15-mal zitiert (Crossref) und nennt 77 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) demonstrate an impressive ability to internalize knowledge and answer natural language questions. Although previous studies validate that LLMs perform well on general knowledge while presenting poor performance on long-tail nuanced knowledge, the community is still doubtful about whether the traditional knowledge graphs should be replaced by LLMs. In this paper, we ask if the schema of knowledge graph (i.e., taxonomy) is made obsolete by LLMs. Intuitively, LLMs should perform well on common taxonomies and at taxonomy levels that are common to people. Unfortunately, there lacks a comprehensive benchmark that evaluates the LLMs over a wide range of taxonomies from common to specialized domains and at levels from root to leaf so that we can draw a confident conclusion. To narrow the research gap, we constructed a novel taxonomy hierarchical structure discovery benchmark named TaxoGlimpse to evaluate the performance of LLMs over taxonomies. TaxoGlimpse covers ten representative taxonomies from common to specialized domains with in-depth experiments of different levels of entities in this taxonomy from root to leaf. Our comprehensive experiments of eighteen LLMs under three prompting settings validate that LLMs perform miserably poorly in handling specialized taxonomies and leaf-level entities. Specifically, the QA accuracy of the best LLM drops by up to 30% as we go from common to specialized domains and from root to leaf levels of taxonomies.

Zitieren

APA

Sun, Y., Xin, H., Sun, K., Xu, Y. E., Yang, X., Dong, X. L., Tang, N., & Chen, L. (2024). Are Large Language Models a Good Replacement of Taxonomies?. Proceedings of the VLDB Endowment, 17(11), 2919-2932. https://doi.org/10.14778/3681954.3681973

Vancouver

Sun Y, Xin H, Sun K, Xu YE, Yang X, Dong XL, et al. Are Large Language Models a Good Replacement of Taxonomies?. Proceedings of the VLDB Endowment. 2024;17(11):2919-2932. doi:10.14778/3681954.3681973

BibTeX
@article{Sun2024,
  author = {Sun, Yushi and Xin, Hao and Sun, Kai and Xu, Yifan Ethan and Yang, Xiao and Dong, Xin Luna and Tang, Nan and Chen, Lei},
  title = {Are Large Language Models a Good Replacement of Taxonomies?},
  journal = {Proceedings of the VLDB Endowment},
  year = {2024},
  volume = {17},
  number = {11},
  pages = {2919--2932},
  doi = {10.14778/3681954.3681973}
}
Kostenlos starten →