A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solving
Proceedings of the AAAI Conference on Artificial Intelligence · 14.03.2026 · 40 (30) · 25226-25234
DOI: 10.1609/aaai.v40i30.39714 ↗Kurz gefasst
Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence, online erschienen am 14.03.2026. Verfasst von Timo Pierre Schrader und 4 weiteren Autor:innen.
Abstract (Originalsprache)
The rise of large language models (LLMs) has sparked interest in coding assistants. While general-purpose programming languages are well supported, generating code for domain-specific languages remains a challenging problem for LLMs. In this paper, we focus on the LLM-based generation of code for Answer Set Programming (ASP), a particularly effective approach for finding solutions to combinatorial search problems. The effectiveness of LLMs in ASP code generation is currently hindered by the limited number of examples seen during their initial pre-training phase. In this paper, we introduce a novel ASP-solver-in-the-loop approach for solver-guided instruction-tuning of LLMs to addressing the highly complex semantic parsing task inherent in ASP code generation. Our method only requires problem specifications in natural language and their solutions. Specifically, we sample ASP statements for program continuations from LLMs for unriddling logic puzzles. Leveraging the special property of declarative ASP programming that partial encodings increasingly narrow down the solution space, we categorize them into chosen and rejected instances based on solver feedback. We then apply supervised fine-tuning to train LLMs on the curated data and further improve robustness using a solver-guided search that includes best-of-N sampling. Our experiments demonstrate consistent improvements in two distinct prompting settings on two datasets.
Zitieren
APA
Schrader, T. P., Lange, L., Kaminski, T., Razniewski, S., & Friedrich, A. (2026). A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solving. Proceedings of the AAAI Conference on Artificial Intelligence, 40(30), 25226-25234. https://doi.org/10.1609/aaai.v40i30.39714
Vancouver
Schrader TP, Lange L, Kaminski T, Razniewski S, Friedrich A. A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solving. Proceedings of the AAAI Conference on Artificial Intelligence. 2026;40(30):25226-25234. doi:10.1609/aaai.v40i30.39714
BibTeX
@article{Schrader2026,
author = {Schrader, Timo Pierre and Lange, Lukas and Kaminski, Tobias and Razniewski, Simon and Friedrich, Annemarie},
title = {A Solver-in-the-Loop Framework for Improving LLMs on Answer Set Programming for Logic Puzzle Solving},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
year = {2026},
volume = {40},
number = {30},
pages = {25226--25234},
doi = {10.1609/aaai.v40i30.39714}
}
EUVIMED