Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models
ACM Transactions on Interactive Intelligent Systems · 30.09.2026 · 16 (3) · 1-23
DOI: 10.1145/3816700 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Interactive Intelligent Systems, online erschienen am 17.08.2026. Verfasst von Reza Shahriari und 6 weiteren Autor:innen. Die Arbeit nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Defense Advanced Research Projects Agency.
Abstract (Originalsprache)
Human-in-the-loop methods leverage human feedback to enhance machine learning and AI. Manual review of outputs can correct errors, identify model weaknesses, or expand labels to broaden model capabilities. Feedback collection methods range from simple flagging of outputs as correct or incorrect to more complex feature-level adjustments or natural language interpretations. This article presents a user study evaluating changes in user performance over time and explores the tradeoff between feedback quality and human effort. We compare four interactive input methods for reviewing and correcting outcomes in object detection and activity recognition in videos. Our findings indicate that while some complex input methods, such as free-text, require more time, the quality and impact of their feedback on model accuracy often surpass those of simpler methods that require less effort. However, more effort does not always lead to better-quality feedback, especially when aiming to improve the model. Our VLM experiments show that the most accurate models were trained using detailed natural language feedback or precise word-level corrections, while simple yes/no judgments also led to solid performance at a much lower annotation cost.
Förderung
- Defense Advanced Research Projects Agency
Zitieren
APA
Shahriari, R., Hashky, A., Arya, S., Audino, T., Ragan, E. D., Gogate, V., & Ruiz, J. (2026). Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models. ACM Transactions on Interactive Intelligent Systems, 16(3), 1-23. https://doi.org/10.1145/3816700
Vancouver
Shahriari R, Hashky A, Arya S, Audino T, Ragan ED, Gogate V, et al. Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models. ACM Transactions on Interactive Intelligent Systems. 2026;16(3):1-23. doi:10.1145/3816700
BibTeX
@article{Shahriari2026,
author = {Shahriari, Reza and Hashky, Amal and Arya, Shivvrat and Audino, Tyler and Ragan, Eric D. and Gogate, Vibhav and Ruiz, Jaime},
title = {Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models},
journal = {ACM Transactions on Interactive Intelligent Systems},
year = {2026},
volume = {16},
number = {3},
pages = {1--23},
doi = {10.1145/3816700}
}
EUVIMED