Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs
ACM Transactions on Computer-Human Interaction · 30.04.2026 · 33 (2) · 1-32
DOI: 10.1145/3793673 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Computer-Human Interaction, online erschienen am 15.04.2026. Verfasst von Mengxi Zhang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: National Natural Science Foundation of China, China University of Geosciences.
Abstract (Originalsprache)
User interactions with mobile applications (apps) are accompanied by continuous visual changes in the Graphical UI (GUI), guiding task completion and feedback. These changes help users complete intended tasks or assess the appropriateness of their actions, typically conveyed through visual cues such as appearance and color. While such visual changes are effective for sighted users, they are inaccessible to blind users, creating substantial barriers to GUI interaction. To address these challenges, we propose VisualDroid , a method based on a multi-modal large language model (LLM) for testing and classifying GUI visual changes using a tailored three-hop reasoning prompting framework. VisualDroid achieved an F1 score of 94.7% in 34 apps from 17 domains, surpassing all baseline methods. When evaluated on five open source apps from F-Droid, our method enabled developers to resolve three identified issues, with two still under review. In terms of efficiency and cost, our method indicates minimal resource consumption.
Förderung
- National Natural Science Foundation of China
- China University of Geosciences
Zitieren
APA
Zhang, M., Yu, J., Xu, C., Li, J., Yin, X., & Liu, H. (2026). Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs. ACM Transactions on Computer-Human Interaction, 33(2), 1-32. https://doi.org/10.1145/3793673
Vancouver
Zhang M, Yu J, Xu C, Li J, Yin X, Liu H. Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs. ACM Transactions on Computer-Human Interaction. 2026;33(2):1-32. doi:10.1145/3793673
BibTeX
@article{Zhang2026,
author = {Zhang, Mengxi and Yu, Jianlin and Xu, Chen and Li, Jiqun and Yin, Xinglong and Liu, Huaxiao},
title = {Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs},
journal = {ACM Transactions on Computer-Human Interaction},
year = {2026},
volume = {33},
number = {2},
pages = {1--32},
doi = {10.1145/3793673}
}
EUVIMED