EUVIMEDEUVIMED entdecken
Crossref

Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia

Stefano Marchesin, Gianmaria Silvello, Omar Alonso

Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · 14.10.2024 · 12 · 105-114

DOI: 10.1609/hcomp.v12i1.31605 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, online erschienen am 14.10.2024. Verfasst von Stefano Marchesin und 2 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Knowledge Graphs (KGs) are essential for applications like search, recommendation, and virtual assistants, where their accuracy directly impacts effectiveness. However, due to their large-scale and ever-evolving nature, it is impractical to manually evaluate all KG contents. We propose a framework that employs sampling, estimation, and active learning to audit KG accuracy in a cost-effective manner. The framework prioritizes KG facts based on their utility to downstream tasks. We applied the framework to DBpedia and gathered annotations from both expert and layman annotators. We also explored the potential of Large Language Models (LLMs) as KG evaluators, showing that while they can perform comparably to low-quality human annotators, they tend to overestimate KG accuracy. As such, LLMs are currently insufficient to replace human crowdworkers in the evaluation process. The results also provide insights into the scalability of methods for auditing KGs.

Zitieren

APA

Marchesin, S., Silvello, G., & Alonso, O. (2024). Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 12, 105-114. https://doi.org/10.1609/hcomp.v12i1.31605

Vancouver

Marchesin S, Silvello G, Alonso O. Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia. Proceedings of the AAAI Conference on Human Computation and Crowdsourcing. 2024;12:105-114. doi:10.1609/hcomp.v12i1.31605

BibTeX
@article{Marchesin2024,
  author = {Marchesin, Stefano and Silvello, Gianmaria and Alonso, Omar},
  title = {Utility-Oriented Knowledge Graph Accuracy Estimation with Limited Annotations: A Case Study on DBpedia},
  journal = {Proceedings of the AAAI Conference on Human Computation and Crowdsourcing},
  year = {2024},
  volume = {12},
  pages = {105--114},
  doi = {10.1609/hcomp.v12i1.31605}
}
Kostenlos starten →