Uncertainty-Aware Vision–Language Design-to-Code Agents with Iterative Design Critique and Structural Fidelity Evaluation
Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online) · 25.06.2026 · 5 (2) · 65-78
DOI: 10.60087/jklst.vol5.n2.005 ↗Kurz gefasst
Fachartikel in der Zeitschrift Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online), online erschienen am 25.06.2026. Verfasst von Elaine Lu. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Design-to-code agents must recover visible content, infer latent webpage structure, generate executable HTML and CSS, and recognize when the resulting page is unreliable. This study evaluated an uncertainty-aware agent on all 484 Design2Code webpages. Three GPT-4V conditions—direct generation, text-cue generation, and critique-and-repair—supplied one candidate per page, yielding 1,452 code artifacts and complete browser renders. A CSS-grouped five-fold protocol prevented related style families from crossing fitting and evaluation partitions. Within each fold, a retrieval-render selector ranked the three candidates, and cross-fitted isotonic regression calibrated its confidence for low-confidence clarification. Fidelity was measured through render success, CLIP similarity, rendered-text agreement, block match, position similarity, CIEDE2000 color similarity, and a deterministic DOM composite combining tag overlap, tag-sequence longest-common-subsequence similarity, node-count similarity, and depth similarity. The repair condition gave a language-model design critic the target screenshot, current render, current HTML, and expected visible-text list before one bounded edit. The evaluation compared generation modes, measured iterative gains, and assessed expected calibration error, Brier score, and risk–coverage behavior. The selector achieved extended fidelity of 0.735738, compared with 0.718614 for direct generation, while calibration reduced ECE from 0.592147 to 0.026508. Browser execution was therefore treated as a necessary condition rather than sufficient evidence, connecting multimodal fidelity directly to selection, repair, and clarification decisions.
Zitieren
APA
Lu, E. (2026). Uncertainty-Aware Vision–Language Design-to-Code Agents with Iterative Design Critique and Structural Fidelity Evaluation. Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online), 5(2), 65-78. https://doi.org/10.60087/jklst.vol5.n2.005
Vancouver
Lu E. Uncertainty-Aware Vision–Language Design-to-Code Agents with Iterative Design Critique and Structural Fidelity Evaluation. Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online). 2026;5(2):65-78. doi:10.60087/jklst.vol5.n2.005
BibTeX
@article{Lu2026,
author = {Lu, Elaine},
title = {Uncertainty-Aware Vision–Language Design-to-Code Agents with Iterative Design Critique and Structural Fidelity Evaluation},
journal = {Journal of Knowledge Learning and Science Technology ISSN: 2959-6386 (online)},
year = {2026},
volume = {5},
number = {2},
pages = {65--78},
doi = {10.60087/jklst.vol5.n2.005}
}
EUVIMED