MIRA: An LLM-Driven Dual-Loop Architecture for Metacognitive Reward Design
Systems · 16.12.2025 · 13 (12) · 1124
DOI: 10.3390/systems13121124 ↗Kurz gefasst
Fachartikel in der Zeitschrift Systems, online erschienen am 16.12.2025. Verfasst von Weiying Zhang und 2 weiteren Autor:innen. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 53 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: National Natural Science Foundation of China, Jiangsu Agriculture Science and Technology Innovation Fund, Natural Science Research Start-up Foundation for Recruiting Talents of Nanjing University.
Abstract (Originalsprache)
A central obstacle to the practical deployment of Reinforcement Learning (RL) is the prevalence of sparse rewards, which often necessitates task-specific dense signals crafted through costly trial-and-error. Automated reward decomposition and return–redistribution methods can reduce this burden, but they are largely semantically agnostic and may fail to capture the multifaceted nature of task performance, leading to reward hacking or stalled exploration. Recent work uses Large Language Models (LLMs) to generate reward functions from high-level task descriptions, but these specifications are typically static and may encode biases or inaccuracies from the pretrained model, resulting in a priori reward misspecification. To address this, we propose the Metacognitive Introspective Reward Architecture (MIRA), a closed-loop architecture that treats LLM-generated reward code as a dynamic object refined through empirical feedback. An LLM first produces a set of computable reward factors. A dual-loop design then decouples policy learning from reward revision: an inner loop jointly trains the agent’s policy and a reward-synthesis network to align with sparse ground-truth outcomes, while an outer loop monitors learning dynamics via diagnostic metrics and, upon detecting pathological signatures, invokes the LLM to perform targeted structural edits. Experiments on MuJoCo benchmarks show that MIRA corrects flawed initial specifications and improves asymptotic performance and sample efficiency over strong reward-design baselines.
Förderung
- National Natural Science Foundation of China
- Jiangsu Agriculture Science and Technology Innovation Fund
- Natural Science Research Start-up Foundation for Recruiting Talents of Nanjing University
- Guizhou Provincial Key Technology R&D Program
Zitieren
APA
Zhang, W., Xu, Y., & Sun, Z. (2025). MIRA: An LLM-Driven Dual-Loop Architecture for Metacognitive Reward Design. Systems, 13(12), 1124. https://doi.org/10.3390/systems13121124
Vancouver
Zhang W, Xu Y, Sun Z. MIRA: An LLM-Driven Dual-Loop Architecture for Metacognitive Reward Design. Systems. 2025;13(12):1124. doi:10.3390/systems13121124
BibTeX
@article{Zhang2025,
author = {Zhang, Weiying and Xu, Yuhua and Sun, Zhixin},
title = {MIRA: An LLM-Driven Dual-Loop Architecture for Metacognitive Reward Design},
journal = {Systems},
year = {2025},
volume = {13},
number = {12},
pages = {1124},
doi = {10.3390/systems13121124}
}
EUVIMED