Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia
Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · 14.10.2024 · 12 · 105-114
DOI: 10.1609/hcomp.v12i1.31605 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, online erschienen am 14.10.2024. Verfasst von Stefano Marchesin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref).
Abstract (Originalsprache)
Knowledge Graphs (KGs) are essential for applications like search, recommendation, and virtual assistants, where their accuracy directly impacts effectiveness. However, due to their large-scale and ever-evolving nature, it is impractical to manually evaluate all KG contents. We propose a framework that employs sampling, estimation, and active learning to audit KG accuracy in a cost-effective manner. The framework prioritizes KG facts based on their utility to downstream tasks. We applied the framework to DBpedia and gathered annotations from both expert and layman annotators. We also explored the potential of Large Language Models (LLMs) as KG evaluators, showing that while they can perform comparably to low-quality human annotators, they tend to overestimate KG accuracy. As such, LLMs are currently insufficient to replace human crowdworkers in the evaluation process. The results also provide insights into the scalability of methods for auditing KGs.
Zitieren
APA
Marchesin, S., Silvello, G., & Alonso, O. (2024). Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 12, 105-114. https://doi.org/10.1609/hcomp.v12i1.31605
Vancouver
Marchesin S, Silvello G, Alonso O. Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2024;12:105-114. doi:10.1609/hcomp.v12i1.31605
BibTeX
@article{Marchesin2024,
author = {Marchesin, Stefano and Silvello, Gianmaria and Alonso, Omar},
title = {Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia},
journal = {Proceedings of the AAAI Conference on Human Computation and Crowdsourcing},
year = {2024},
volume = {12},
pages = {105--114},
doi = {10.1609/hcomp.v12i1.31605}
}
EUVIMED