A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions
Journal of Artificial Intelligence Research · 26.03.2026 · 85
DOI: 10.1613/jair.1.19490 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Artificial Intelligence Research, online erschienen am 26.03.2026. Verfasst von Pascal J. Sager und 9 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref).
Abstract (Originalsprache)
Background
Agents for computer use (ACUs) are systems that execute complex tasks on digital devices – such as personal computers or mobile phones – given instructions in natural language. These agents automate tasks by controlling software through low-level actions like mouse clicks and touchscreen gestures. However, despite rapid progress, ACUs are not yet mature for everyday use.
Objectives
This survey examines the current state-of-the-art, identifies trends, and points out research gaps in the development of practical ACUs. The goal is to provide a comprehensive review and analysis that helps advance general-purpose, robust, and scalable agents for real-world computer use.
Methods
We introduce a multifaceted taxonomy of ACUs across three dimensions: (I) the domain perspective, characterizing the contexts in which agents operate; (II) the interaction perspective, describing observation modalities (e.g., screenshots, HTML) and action modalities (e.g., mouse, keyboard, code execution); and (III) the agent perspective, detailing how agents perceive, reason, and learn. We review 87 original research papers about ACUs and 33 relevant datasets, covering both foundation model-based and specialized approaches.
Results
Our taxonomy comprehensively structures state-of-the-art approaches and establishes the groundwork for guiding future ACU research. We found that the field is transitioning from specialized agents toward foundation-model-based agents, a shift from text to image-based observation space, and an increasing adoption of behavior cloning methodologies. Furthermore, we identify six key research gaps: insufficient generalization, inefficient learning, limited planning, low task complexity in benchmarks, non-standardized evaluation, and a disconnect between research and practical conditions.
Conclusions
To continue rapid improvements in the field, we recommend focusing on: (a) vision-based observations and low-level control to enhance generalization; (b) adaptive learning beyond static prompting; (c) effective planning and reasoning capabilities; (d) realistic, high-complexity benchmarks; (e) standardized evaluation criteria based on task success; and (f) aligning agent design with real-world deployment constraints. Collectively, our findings and proposed directions help develop more general-purpose agents for everyday digital tasks.
Zitieren
APA
Sager, P. J., Meyer, B., Yan, P., Wartburg-Kottler, R. V., Etaiwi, L., Enayati, A., Nobel, G., Abdulkadir, A., Grewe, B. F., & Stadelmann, T. (2026). A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions. Journal of Artificial Intelligence Research, 85. https://doi.org/10.1613/jair.1.19490
Vancouver
Sager PJ, Meyer B, Yan P, Wartburg-Kottler RV, Etaiwi L, Enayati A, et al. A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions. Journal of Artificial Intelligence Research. 2026;85. doi:10.1613/jair.1.19490
BibTeX
@article{Sager2026,
author = {Sager, Pascal J. and Meyer, Benjamin and Yan, Peng and Wartburg-Kottler, Rebekka Von and Etaiwi, Layan and Enayati, Aref and Nobel, Gabriel and Abdulkadir, Ahmed and Grewe, Benjamin F. and Stadelmann, Thilo},
title = {A Comprehensive Survey of Agents for Computer Use: Foundations, Challenges, and Future Directions},
journal = {Journal of Artificial Intelligence Research},
year = {2026},
volume = {85},
doi = {10.1613/jair.1.19490}
}
EUVIMED