EUVIMEDEUVIMED entdecken
Crossref

Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs

Mengxi Zhang, Jianlin Yu, Chen Xu, Jiqun Li, Xinglong Yin, Huaxiao Liu

ACM Transactions on Computer-Human Interaction · 30.04.2026 · 33 (2) · 1-32

DOI: 10.1145/3793673 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Computer-Human Interaction, online erschienen am 15.04.2026. Verfasst von Mengxi Zhang und 5 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 68 Literaturangaben. Gefördert von: National Natural Science Foundation of China, China University of Geosciences.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

User interactions with mobile applications (apps) are accompanied by continuous visual changes in the Graphical UI (GUI), guiding task completion and feedback. These changes help users complete intended tasks or assess the appropriateness of their actions, typically conveyed through visual cues such as appearance and color. While such visual changes are effective for sighted users, they are inaccessible to blind users, creating substantial barriers to GUI interaction. To address these challenges, we propose VisualDroid , a method based on a multi-modal large language model (LLM) for testing and classifying GUI visual changes using a tailored three-hop reasoning prompting framework. VisualDroid achieved an F1 score of 94.7% in 34 apps from 17 domains, surpassing all baseline methods. When evaluated on five open source apps from F-Droid, our method enabled developers to resolve three identified issues, with two still under review. In terms of efficiency and cost, our method indicates minimal resource consumption.

Förderung

  • National Natural Science Foundation of China
  • China University of Geosciences

Zitieren

APA

Zhang, M., Yu, J., Xu, C., Li, J., Yin, X., & Liu, H. (2026). Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs. ACM Transactions on Computer-Human Interaction, 33(2), 1-32. https://doi.org/10.1145/3793673

Vancouver

Zhang M, Yu J, Xu C, Li J, Yin X, Liu H. Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs. ACM Transactions on Computer-Human Interaction. 2026;33(2):1-32. doi:10.1145/3793673

BibTeX
@article{Zhang2026,
  author = {Zhang, Mengxi and Yu, Jianlin and Xu, Chen and Li, Jiqun and Yin, Xinglong and Liu, Huaxiao},
  title = {Towards Testing the Accessibility of Dynamic Visual Changes in Android Mobile GUI with Multi-Modal LLMs},
  journal = {ACM Transactions on Computer-Human Interaction},
  year = {2026},
  volume = {33},
  number = {2},
  pages = {1--32},
  doi = {10.1145/3793673}
}
Kostenlos starten →