EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

ORKG-Leaderboards: a systematic workflow for mining leaderboards as a knowledge graph

Salomon Kabongo, Jennifer D’Souza, Sören Auer

International Journal on Digital Libraries · 2024-03 · 25 (1) · 41-54

DOI: 10.1007/s00799-023-00366-1 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal on Digital Libraries, online erschienen am 15.06.2023. Verfasst von Salomon Kabongo und 2 weiteren Autor:innen. Die Arbeit wurde bisher 9-mal zitiert (Crossref) und nennt 44 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Bundesministerium für Bildung und Forschung, FP7 Ideas: European Research Council.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The purpose of this work is to describe the orkg -Leaderboard software designed to extract leaderboards defined as task–dataset–metric tuples automatically from large collections of empirical research papers in artificial intelligence (AI). The software can support both the main workflows of scholarly publishing, viz. as LaTeX files or as PDF files. Furthermore, the system is integrated with the open research knowledge graph (ORKG) platform, which fosters the machine-actionable publishing of scholarly findings. Thus, the systemsss output, when integrated within the ORKG’s supported Semantic Web infrastructure of representing machine-actionable ‘resources’ on the Web, enables: (1) broadly, the integration of empirical results of researchers across the world, thus enabling transparency in empirical research with the potential to also being complete contingent on the underlying data source(s) of publications; and (2) specifically, enables researchers to track the progress in AI with an overview of the state-of-the-art across the most common AI tasks and their corresponding datasets via dynamic ORKG frontend views leveraging tables and visualization charts over the machine-actionable data. Our best model achieves performances above 90% F1 on the leaderboard extraction task, thus proving orkg -Leaderboards a practically viable tool for real-world usage. Going forward, in a sense, orkg -Leaderboards transforms the leaderboard extraction task to an automated digitalization task, which has been, for a long time in the community, a crowdsourced endeavor.

Förderung

  • Bundesministerium für Bildung und Forschung
  • FP7 Ideas: European Research Council

Zitieren

APA

Kabongo, S., D’Souza, J., & Auer, S. (2024). ORKG-Leaderboards: a systematic workflow for mining leaderboards as a knowledge graph. International Journal on Digital Libraries, 25(1), 41-54. https://doi.org/10.1007/s00799-023-00366-1

Vancouver

Kabongo S, D’Souza J, Auer S. ORKG-Leaderboards: a systematic workflow for mining leaderboards as a knowledge graph. International Journal on Digital Libraries. 2024;25(1):41-54. doi:10.1007/s00799-023-00366-1

BibTeX
@article{Kabongo2024,
  author = {Kabongo, Salomon and D’Souza, Jennifer and Auer, Sören},
  title = {ORKG-Leaderboards: a systematic workflow for mining leaderboards as a knowledge graph},
  journal = {International Journal on Digital Libraries},
  year = {2024},
  volume = {25},
  number = {1},
  pages = {41--54},
  doi = {10.1007/s00799-023-00366-1}
}
Kostenlos starten →