Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (40) · 33782-33790
DOI: 10.1609/aaai.v40i40.40669 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Zijian Wang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref).
Abstract (Originalsprache)
Chain-of-Thought (CoT) reasoning is a critical capability for large language models (LLMs), enabling them to tackle complex multi-step tasks. While base LLMs, pre-trained on general text corpora, often struggle with reasoning due to a lack of specialized training, recent studies reveal their latent reasoning potential tied to hidden states. However, existing hidden state manipulation methods, such as linear activation steering, suffer from limitations due to their rigid and unconstrained nature, often leading to distribution shifts and degraded text quality. In this work, we propose a novel approach for eliciting CoT reasoning from base LLMs through hidden state manipulation grounded in probabilistic conditional generation. By reformulating the challenge as an optimization problem with a balanced likelihood and prior regularization framework, our method guides hidden states toward reasoning-oriented trajectories while preserving linguistic coherence. Extensive evaluations across mathematical, commonsense, and logical reasoning benchmarks demonstrate that our approach consistently outperforms existing steering methods, offering a theoretically principled and effective solution for enhancing reasoning capabilities in base LLMs.
Zitieren
APA
Wang, Z., Ma, Y., & Xu, C. (2026). Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40), 33782-33790. https://doi.org/10.1609/aaai.v40i40.40669
Vancouver
Wang Z, Ma Y, Xu C. Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(40):33782-33790. doi:10.1609/aaai.v40i40.40669
BibTeX
@article{Wang2026,
author = {Wang, Zijian and Ma, Yanxiang and Xu, Chang},
title = {Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {40},
pages = {33782--33790},
doi = {10.1609/aaai.v40i40.40669}
}
EUVIMED