EUVIMEDEUVIMED entdecken
Crossref

Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim, Sundong Kim

ACM Transactions on Intelligent Systems and Technology · 31.12.2025 · 16 (6) · 1-52

DOI: 10.1145/3712701 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Intelligent Systems and Technology, online erschienen am 24.11.2025. Verfasst von Seungpil Lee und 8 weiteren Autor:innen. Die Arbeit wurde bisher 30-mal zitiert (Crossref) und nennt 81 Literaturangaben. Gefördert von: Accelerate Foundation Models Research Initiative, Development of Artificial Complex Intelligence for Conceptually Understanding and Inferring Like Human, Testbed for Abstraction and Reasoning.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

The existing methods for evaluating the inference abilities of Large Language Models (LLMs) have been predominantly results-centric, making it challenging to assess the inference process comprehensively. We introduce a novel approach using the Abstraction and Reasoning Corpus (ARC) benchmark to evaluate the inference and contextual understanding abilities of LLMs in a process-centric manner, focusing on three key components from the Language of Thought Hypothesis (LoTH): Logical Coherence, Compositionality, and Productivity. Our carefully designed experiments reveal that while LLMs demonstrate some inference capabilities, they still significantly lag behind human-level reasoning in these three aspects. The main contribution of this article lies in introducing the LoTH perspective, which provides a method for evaluating the reasoning process that conventional results-oriented approaches fail to capture, thereby offering new insights into the development of human-level reasoning in artificial intelligence systems.

Förderung

  • Accelerate Foundation Models Research Initiative
  • Development of Artificial Complex Intelligence for Conceptually Understanding and Inferring Like Human
  • Testbed for Abstraction and Reasoning

Zitieren

APA

Lee, S., Sim, W., Shin, D., Seo, W., Park, J., Lee, S., Hwang, S., Kim, S., & Kim, S. (2025). Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus. ACM Transactions on Intelligent Systems and Technology, 16(6), 1-52. https://doi.org/10.1145/3712701

Vancouver

Lee S, Sim W, Shin D, Seo W, Park J, Lee S, et al. Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus. ACM Transactions on Intelligent Systems and Technology. 2025;16(6):1-52. doi:10.1145/3712701

BibTeX
@article{Lee2025,
  author = {Lee, Seungpil and Sim, Woochang and Shin, Donghyeon and Seo, Wongyu and Park, Jiwon and Lee, Seokki and Hwang, Sanha and Kim, Sejin and Kim, Sundong},
  title = {Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus},
  journal = {ACM Transactions on Intelligent Systems and Technology},
  year = {2025},
  volume = {16},
  number = {6},
  pages = {1--52},
  doi = {10.1145/3712701}
}
Kostenlos starten →