EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Recognition of GUI Web Elements using the Deep Learning Approach and Yolo Architecture

Helga Prifti, Bekir Karlik

Systems and Computing · 2026

DOI: 10.64409/sycom.v2.i1.28 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Systems and Computing, 2026. Verfasst von Helga Prifti und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Context

The growing complexity of modern web and mobile applications, shaped by rapid UI/UX advancements and agile software development practices, demands efficient and automated recognition of graphical user interface (GUI) elements, as manual testing remains costly and time-consuming.

Objective

This study aims to evaluate the performance of the YOLOv8 deep learning architecture for accurate recognition of diverse GUI components, such as buttons, fields, headings, links, and images, from real-world application screenshots.

Method

A small YOLOv8 model was trained on the Roboflow GUI element detection dataset containing over 1,000 annotated website screenshots, using 35 training epochs, a batch size of 8, and an image resolution of 640×640.

Results

The model achieved measurable improvements, with precision rising from 0.368 to 0.454, recall increasing from 0.296 to 0.425, and mAP50–95 improving from 0.101 to 0.232. Strongest detection was observed for buttons and input fields, while weaker performance was noted for iframes and labels due to their inherent ambiguity.

Conclusions

The results demonstrate that YOLOv8 has significant potential in automating GUI recognition, reducing reliance on manual testing in agile workflows, and improving interface validation. Further optimization with larger datasets and advanced augmentation methods is recommended to enhance robustness and generalization

Zitieren

APA

Prifti, H., & Karlik, B. (2026). Recognition of GUI Web Elements using the Deep Learning Approach and Yolo Architecture. Systems and Computing. https://doi.org/10.64409/sycom.v2.i1.28

Vancouver

Prifti H, Karlik B. Recognition of GUI Web Elements using the Deep Learning Approach and Yolo Architecture. Systems and Computing. 2026. doi:10.64409/sycom.v2.i1.28

BibTeX
@article{Prifti2026,
  author = {Prifti, Helga and Karlik, Bekir},
  title = {Recognition of GUI Web Elements using the Deep Learning Approach and Yolo Architecture},
  journal = {Systems and Computing},
  year = {2026},
  doi = {10.64409/sycom.v2.i1.28}
}
Kostenlos starten →