EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Behavior and representation in open-weight Large Language Models for combinatorial optimization: From feature extraction to algorithm selection

Francesca Da Ros, Luca Di Gaspero, Kevin Roitero

Computers & Operations Research · 2026-12 · 196 · 107603

DOI: 10.1016/j.cor.2026.107603 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Computers & Operations Research, 2026-12. Verfasst von Francesca Da Ros und 2 weiteren Autor:innen. Die Arbeit nennt 66 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Regione Autonoma Friuli Venezia Giulia, Ministero dell'Università e della Ricerca.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Recent advances in Large Language Models (LLMs) open new perspectives for automation in optimization, yet little is known about whether their internal representations capture problem structure or algorithmic behavior. We investigate whether representations learned by frozen, open-weight LLMs for combinatorial optimization instances can support downstream decision tasks. The goal is not to replace exact feature extractors or to propose a new algorithm, but to assess whether such representations are reusable for feature recovery and algorithm selection. Our methodology combines direct querying, which tests explicit feature extraction, with probing analyses of whether this information is implicitly encoded in the hidden layers. The probing framework is further extended to a per-instance algorithm selection task. Experiments span four benchmark problems, three instance representations, and five open-weight models from 3B to 120B parameters across the Llama instruct and GPT reasoning families, including a chain-of-thought study on Llama models. Results show a limited ability to recover features explicitly, particularly those requiring structured computation, while part of this information remains implicitly encoded in the hidden states. A consistent gap separates implicit encoding from explicit retrieval. Model scale attenuates this gap, whereas the effect of chain-of-thought prompting depends strongly on model size, and in all cases the gap remains open. Reasoning-oriented models also tend to abstain more when reliable computation is not possible. Notably, the predictive power of LLM hidden-layer representations is comparable to traditional feature extraction across all five models, indicating they can act as effective surrogate descriptors for downstream optimization tasks.

Förderung

  • Regione Autonoma Friuli Venezia Giulia
  • Ministero dell'Università e della Ricerca

Zitieren

APA

Ros, F. D., Gaspero, L. D., & Roitero, K. (2026). Behavior and representation in open-weight Large Language Models for combinatorial optimization: From feature extraction to algorithm selection. Computers & Operations Research, 196, 107603. https://doi.org/10.1016/j.cor.2026.107603

Vancouver

Ros FD, Gaspero LD, Roitero K. Behavior and representation in open-weight Large Language Models for combinatorial optimization: From feature extraction to algorithm selection. Computers & Operations Research. 2026;196:107603. doi:10.1016/j.cor.2026.107603

BibTeX
@article{Ros2026,
  author = {Ros, Francesca Da and Gaspero, Luca Di and Roitero, Kevin},
  title = {Behavior and representation in open-weight Large Language Models for combinatorial optimization: From feature extraction to algorithm selection},
  journal = {Computers & Operations Research},
  year = {2026},
  volume = {196},
  pages = {107603},
  doi = {10.1016/j.cor.2026.107603}
}
Kostenlos starten →