EUVIMEDEUVIMED entdecken
Crossref

OmniBench: A Comprehensive Benchmark Integrating Real-World, Time-sensitive, and Multi-Hop Questions with a Multi-Dimensional Hybrid Evaluation Framework

Wenjie Wang, Yufeng Jiang, Ge Sun, Chenghang Dong, Zheng Jun, Li Mengjie, Lixin Chen, Huan Wang, Haoyu Wang, Bin Chen

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 33657-33665

DOI: 10.1609/aaai.v40i40.40655 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Wenjie Wang und 9 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recently, with the increasing capabilities of Large Language Models (LLMs), AI applications have gradually emerged to solve various problems in people's daily lives, so accurately measuring their performance and reliability is paramount. However, existing benchmarks predominantly rely on closed-ended, multiple-choice or short-answer question formats. While useful for assessment, these formats exhibit a significant gap compared to the diverse and open-ended nature of questions posed by real-world users. To bridge this gap, we produce OmniBench, a comprehensive open-domain benchmark. OmniBench is uniquely composed of authentic, user-generated questions harvested from real-world interactions on various websites and applications, covering 16 rigorously defined knowledge domains and 5 crucial user intents derived from a large-scale analysis of the mass corpus. Crucially, we propose three automated data construction pipelines that enable the continuous and periodic updating of the benchmark dataset. This approach not only ensures that the questions can keep up with current events, but also effectively mitigates the critical issue of data contamination prevalent in static benchmarks. Moreover, a multi-dimensional hybrid evaluation framework named OmniEval is proposed for evaluating the responses. This framework combines diverse metrics and evaluation methods to capture nuanced aspects of answer performance. Extensive validation demonstrates that this evaluation framework exhibits strong alignment with human judgments, ensuring the reliability of the benchmark results.

Zitieren

APA

Wang, W., Jiang, Y., Sun, G., Dong, C., Jun, Z., Mengjie, L., Chen, L., Wang, H., Wang, H., & Chen, B. (2026). OmniBench: A Comprehensive Benchmark Integrating Real-World, Time-sensitive, and Multi-Hop Questions with a Multi-Dimensional Hybrid Evaluation Framework. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 33657-33665. https://doi.org/10.1609/aaai.v40i40.40655

Vancouver

Wang W, Jiang Y, Sun G, Dong C, Jun Z, Mengjie L, et al. OmniBench: A Comprehensive Benchmark Integrating Real-World, Time-sensitive, and Multi-Hop Questions with a Multi-Dimensional Hybrid Evaluation Framework. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):33657-33665. doi:10.1609/aaai.v40i40.40655

BibTeX
@article{Wang2026,
  author = {Wang, Wenjie and Jiang, Yufeng and Sun, Ge and Dong, Chenghang and Jun, Zheng and Mengjie, Li and Chen, Lixin and Wang, Huan and Wang, Haoyu and Chen, Bin},
  title = {OmniBench: A Comprehensive Benchmark Integrating Real-World, Time-sensitive, and Multi-Hop Questions with a Multi-Dimensional Hybrid Evaluation Framework},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {40},
  pages = {33657--33665},
  doi = {10.1609/aaai.v40i40.40655}
}
Kostenlos starten →