EUVIMEDEUVIMED entdecken
Crossref

Evaluating LLM Reasoning in the Operations Research Domain with ORQA

Mahdi Mostajabdaveh, Timothy Tin Long Yu, Samarendra Chandan Bindu Dash, Rindra Ramamonjison, Jabo Serge Byusa, Giuseppe Carenini, Zirui Zhou, Yong Zhang

Proceedings of the AAAI Conference on Artificial Intelligence · 11.04.2025 · 39 (23) · 24902-24910

DOI: 10.1609/aaai.v39i23.34673 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 11.04.2025. Verfasst von Mahdi Mostajabdaveh und 7 weiteren Autor:innen. Die Arbeit wurde bisher 7-mal zitiert (Crossref).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark, to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark is designed to evaluate whether LLMs can emulate the knowledge and reasoning skills of OR experts when given diverse and complex optimization problems. The dataset, crafted by OR experts, presents real-world optimization problems that require multistep reasoning to build their mathematical models. Our evaluations of various open-source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral reveal their modest performance, indicating a gap in their aptitude to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs’ generalization capabilities, providing insights for future research in this area. The dataset and evaluation code are publicly available.

Zitieren

APA

Mostajabdaveh, M., Yu, T. T. L., Dash, S. C. B., Ramamonjison, R., Byusa, J. S., Carenini, G., Zhou, Z., & Zhang, Y. (2025). Evaluating LLM Reasoning in the Operations Research Domain with ORQA. Proceedings of the AAAI Conference on Artificial Intelligence, 39(23), 24902-24910. https://doi.org/10.1609/aaai.v39i23.34673

Vancouver

Mostajabdaveh M, Yu TTL, Dash SCB, Ramamonjison R, Byusa JS, Carenini G, et al. Evaluating LLM Reasoning in the Operations Research Domain with ORQA. Proceedings of the AAAI Conference on Artificial Intelligence. 2025;39(23):24902-24910. doi:10.1609/aaai.v39i23.34673

BibTeX
@article{Mostajabdaveh2025,
  author = {Mostajabdaveh, Mahdi and Yu, Timothy Tin Long and Dash, Samarendra Chandan Bindu and Ramamonjison, Rindra and Byusa, Jabo Serge and Carenini, Giuseppe and Zhou, Zirui and Zhang, Yong},
  title = {Evaluating LLM Reasoning in the Operations Research Domain with ORQA},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2025},
  volume = {39},
  number = {23},
  pages = {24902--24910},
  doi = {10.1609/aaai.v39i23.34673}
}
Kostenlos starten →