EUVIMEDEUVIMED entdecken
Crossref

Augmenting Vision-Language Agents with Spatial Grounding for Gameplay Automation

Martin Mitrović, Niels Justesen, Sebastian Risi, Christoffer Holmgård

Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment · 08.11.2026 · 22 (1) · 331-341

DOI: 10.1609/aiide.v22i1.43591 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, online erschienen am 08.11.2026. Verfasst von Martin Mitrović und 3 weiteren Autor:innen.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Building an autonomous agent capable of playing contemporary video games poses a significant technical challenge, requiring mastery of visual perception, real-time decision-making, long-term memory, and adaptive planning within complex, dynamic environments. While Vision-Language Models (VLMs) demonstrate strong reasoning capabilities, they struggle with spatial grounding, often failing to accurately localize gameplay objects or UI components. This gap between reasoning and perception is a key limitation when VLMs are used as game-playing agents. To address this, we present a framework that augments visual inputs to mitigate spatial grounding errors. These take the form of bounding boxes around game elements, generated using two methods: Segment Identification (SI), which segments each frame and labels the pieces with a VLM online, and a lightweight YOLO detector fine-tuned offline on gameplay footage. We evaluate four VLMs across three casual mobile games unaugmented and under each augmentation, over five trials per configuration. Several models benefit substantially from augmentation, gaining the capability to complete challenges that were out of reach from raw screenshots alone. The benefit is conditional rather than uniform. It improves the models that cannot localize interactive elements on their own, but can interfere with one that already performs well. Our experiments show that the same augmentation substantially improves one model's efficiency while degrading another's. Moreover, on localization-limited tasks, augmentation lets a smaller open model match a much larger proprietary one, though this parity does not extend to challenges bottlenecked by planning. These findings position object-level grounding as a promising but situational tool for VLM-based game agents, clarifying when it helps, hurts, or has no measurable effect.

Zitieren

APA

Mitrović, M., Justesen, N., Risi, S., & Holmgård, C. (2026). Augmenting Vision-Language Agents with Spatial Grounding for Gameplay Automation. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 22(1), 331-341. https://doi.org/10.1609/aiide.v22i1.43591

Vancouver

Mitrović M, Justesen N, Risi S, Holmgård C. Augmenting Vision-Language Agents with Spatial Grounding for Gameplay Automation. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment. 2026;22(1):331-341. doi:10.1609/aiide.v22i1.43591

BibTeX
@article{Mitrovi2026,
  author = {Mitrović, Martin and Justesen, Niels and Risi, Sebastian and Holmgård, Christoffer},
  title = {Augmenting Vision-Language Agents with Spatial Grounding for Gameplay Automation},
  journal = {Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment},
  year = {2026},
  volume = {22},
  number = {1},
  pages = {331--341},
  doi = {10.1609/aiide.v22i1.43591}
}
Kostenlos starten →