EUVIMEDEUVIMED entdecken
Open Access · CC BY-SA 4.0

Grounded AI Pointer Cards for Browser Agents: Evidence-Aware Web Element Reranking and Action Explanation on WebLINX

Yinchen Shi, Yuxuan Ren

International Journal of Graphic Design · 30.04.2026 · 4 (1) · 224-245

DOI: 10.51903/ijgd.v4i1.4002 ↗

Kurz gefasst

Fachartikel in der Zeitschrift International Journal of Graphic Design, online erschienen am 30.04.2026. Verfasst von Yinchen Shi und 1 weiteren Autor:innen. Der Volltext ist frei zugänglich (Open Access, CC BY-SA 4.0).

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

AI browser agents increasingly promise to turn the mouse pointer into a situated interface for selecting page content, asking follow-up questions, scrolling to relevant sections, and opening contextual panels. For graphic design research, this shift matters because the pointer card is both a computational output and a visual communication device: it must make the target, supporting evidence, and next action legible at the moment of use. This paper evaluates evidence-aware web element reranking as the grounding layer for AI pointer cards. The empirical study uses the WebLINX candidate-reranking data, including validation, same-distribution test, and unseen-website test partitions. Each query is paired with candidate DOM/page elements and binary relevance labels. Nine reranking methods were evaluated, ranging from random and DOM-salience controls to BM25 variants, a transparent hybrid formula, a logistic reranker, and a depth-limited decision-tree reranker. Across 5,883 queries and 1,556,655 query–element pairs, the tree reranker achieved the highest same-distribution nDCG@10 (0.272) and Recall@10 (0.473), while the fixed hybrid achieved the highest unseen-website Top-1 accuracy (0.104). Combining lexical and DOM evidence improved ranking, whereas layout features transferred less consistently across websites. Because the first-ranked target was correct in only 10.4% of unseen-website queries, the findings support a confirm-before-act grounding prototype with visible alternatives and correction controls rather than autonomous execution. The reported metrics evaluate target ranking rather than card usability.

Zitieren

APA

Shi, Y., & Ren, Y. (2026). Grounded AI Pointer Cards for Browser Agents: Evidence-Aware Web Element Reranking and Action Explanation on WebLINX. International Journal of Graphic Design, 4(1), 224-245. https://doi.org/10.51903/ijgd.v4i1.4002

Vancouver

Shi Y, Ren Y. Grounded AI Pointer Cards for Browser Agents: Evidence-Aware Web Element Reranking and Action Explanation on WebLINX. International Journal of Graphic Design. 2026;4(1):224-245. doi:10.51903/ijgd.v4i1.4002

BibTeX
@article{Shi2026,
  author = {Shi, Yinchen and Ren, Yuxuan},
  title = {Grounded AI Pointer Cards for Browser Agents: Evidence-Aware Web Element Reranking and Action Explanation on WebLINX},
  journal = {International Journal of Graphic Design},
  year = {2026},
  volume = {4},
  number = {1},
  pages = {224--245},
  doi = {10.51903/ijgd.v4i1.4002}
}
Kostenlos starten →