Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development
ACM Transactions on Computer-Human Interaction · 31.08.2026 · 33 (4) · 1-38
DOI: 10.1145/3817601 ↗Kurz gefasst
Fachartikel in der Zeitschrift ACM Transactions on Computer-Human Interaction, online erschienen am 08.08.2026. Verfasst von Billy Shi und 1 weiteren Autor:innen. Die Arbeit nennt 96 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0).
Abstract (Originalsprache)
Applications of large language models (LLMs) in software engineering are soaring quickly. Despite ample evidence supporting LLMs’ capabilities of generating high-quality and creative suggestions, human–LLM interactions in software testing remain under-explored, and there lacks empirical evidence supporting efficacious human–LLM system designs. In software testing, the industry’s pursuit of “autonomous” systems neglects intensive human involvement in prior-design and post-review processes. This article discusses two empirical user studies for a test case brainstorming task: (1) exploring user behaviors in human–LLM interactions compared to web search ( \(N_{1}=16\) ) and (2) investigating three modified interaction strategies—preemptive prompting, buffered response, and guided input ( \(N_{2}=24\) ). We consolidate nine cross-disciplinary metrics to quantitatively evaluate the holistic performance of the human–LLM system, covering three perspectives: test quality, creativity, and attention span. Our findings reveal that users spend 126% more time interacting with LLMs compared to Google search. Moreover, preemptively prompting the LLM system significantly improves test quality and task creativity by over 30%, while simultaneously reducing user idle time by up to 49%. Based on the results, this article discusses three interaction design principles—mixed initiative, acceptability, and appropriation—as guidance for future iterations of an efficacious LLM-assisted software testing system.
Zitieren
APA
Shi, B., & Kristensson, P. O. (2026). Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development. ACM Transactions on Computer-Human Interaction, 33(4), 1-38. https://doi.org/10.1145/3817601
Vancouver
Shi B, Kristensson PO. Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development. ACM Transactions on Computer-Human Interaction. 2026;33(4):1-38. doi:10.1145/3817601
BibTeX
@article{Shi2026,
author = {Shi, Billy and Kristensson, Per Ola},
title = {Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development},
journal = {ACM Transactions on Computer-Human Interaction},
year = {2026},
volume = {33},
number = {4},
pages = {1--38},
doi = {10.1145/3817601}
}
EUVIMED