EUVIMEDEUVIMED entdecken
Crossref

Interpretable Reward Model via Sparse Autoencoder

Shuyi Zhang, Wei Shi, Sihang Li, Jiayi Liao, Tao Liang, Hengxing Cai, Xiang Wang

Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (41) · 34808-34816

DOI: 10.1609/aaai.v40i41.40783 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Shuyi Zhang und 6 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Large language models (LLMs) have been widely deployed across numerous fields. Reinforcement Learning from Human Feedback (RLHF) leverages reward models (RMs) as proxies for human preferences to align LLM behaviors with human values, making the accuracy, reliability, and interpretability of RMs critical for effective alignment. However, traditional RMs lack interpretability, offer limited insight into the reasoning behind reward assignments, and are inflexible toward user preference shifts. While recent multidimensional RMs aim for improved interpretability, they often fail to provide feature-level attribution and require costly annotations. To overcome these limitations, we introduce the Sparse Autoencoder-Enhanced Reward Model (SARM), a novel architecture that integrates a pretrained Sparse Autoencoder (SAE) into a reward model. SARM maps the hidden activations of LLM-based RM into an interpretable, sparse, and monosemantic feature space, from which a scalar head aggregates feature activations to produce transparent and conceptually meaningful reward scores. Empirical evaluations demonstrate that SARM facilitates direct feature-level attribution of reward assignments, allows dynamic adjustment to preference shifts, and achieves superior alignment performance compared to conventional reward models.

Zitieren

APA

Zhang, S., Shi, W., Li, S., Liao, J., Liang, T., Cai, H., & Wang, X. (2026). Interpretable Reward Model via Sparse Autoencoder. Proceedings of the AAAI Conference on Artificial Intelligence, 40(41), 34808-34816. https://doi.org/10.1609/aaai.v40i41.40783

Vancouver

Zhang S, Shi W, Li S, Liao J, Liang T, Cai H, et al. Interpretable Reward Model via Sparse Autoencoder. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(41):34808-34816. doi:10.1609/aaai.v40i41.40783

BibTeX
@article{Zhang2026,
  author = {Zhang, Shuyi and Shi, Wei and Li, Sihang and Liao, Jiayi and Liang, Tao and Cai, Hengxing and Wang, Xiang},
  title = {Interpretable Reward Model via Sparse Autoencoder},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
  year = {2026},
  volume = {40},
  number = {41},
  pages = {34808--34816},
  doi = {10.1609/aaai.v40i41.40783}
}
Kostenlos starten →