EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models

Reza Shahriari, Amal Hashky, Shivvrat Arya, Tyler Audino, Eric D. Ragan, Vibhav Gogate, Jaime Ruiz

ACM Transactions on Interactive Intelligent Systems · 30.09.2026 · 16 (3) · 1-23

DOI: 10.1145/3816700 ↗

Kurz gefasst

Fachartikel in der Zeitschrift ACM Transactions on Interactive Intelligent Systems, online erschienen am 17.08.2026. Verfasst von Reza Shahriari und 6 weiteren Autor:innen. Die Arbeit nennt 51 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Defense Advanced Research Projects Agency.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Human-in-the-loop methods leverage human feedback to enhance machine learning and AI. Manual review of outputs can correct errors, identify model weaknesses, or expand labels to broaden model capabilities. Feedback collection methods range from simple flagging of outputs as correct or incorrect to more complex feature-level adjustments or natural language interpretations. This article presents a user study evaluating changes in user performance over time and explores the tradeoff between feedback quality and human effort. We compare four interactive input methods for reviewing and correcting outcomes in object detection and activity recognition in videos. Our findings indicate that while some complex input methods, such as free-text, require more time, the quality and impact of their feedback on model accuracy often surpass those of simpler methods that require less effort. However, more effort does not always lead to better-quality feedback, especially when aiming to improve the model. Our VLM experiments show that the most accurate models were trained using detailed natural language feedback or precise word-level corrections, while simple yes/no judgments also led to solid performance at a much lower annotation cost.

Förderung

  • Defense Advanced Research Projects Agency

Zitieren

APA

Shahriari, R., Hashky, A., Arya, S., Audino, T., Ragan, E. D., Gogate, V., & Ruiz, J. (2026). Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models. ACM Transactions on Interactive Intelligent Systems, 16(3), 1-23. https://doi.org/10.1145/3816700

Vancouver

Shahriari R, Hashky A, Arya S, Audino T, Ragan ED, Gogate V, et al. Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models. ACM Transactions on Interactive Intelligent Systems. 2026;16(3):1-23. doi:10.1145/3816700

BibTeX
@article{Shahriari2026,
  author = {Shahriari, Reza and Hashky, Amal and Arya, Shivvrat and Audino, Tyler and Ragan, Eric D. and Gogate, Vibhav and Ruiz, Jaime},
  title = {Comparison of Text-Based Inputs for Human-in-the-Loop Feedback in Vision-Language Models},
  journal = {ACM Transactions on Interactive Intelligent Systems},
  year = {2026},
  volume = {16},
  number = {3},
  pages = {1--23},
  doi = {10.1145/3816700}
}
Kostenlos starten →