EUVIMEDDescoperă EUVIMED
Open access · CC BY 4.0

Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development

Billy Shi, Per Ola Kristensson

ACM Transactions on Computer-Human Interaction · 2026-08-31 · 33 (4) · 1-38

DOI: 10.1145/3817601 ↗

At a glance

Journal article in ACM Transactions on Computer-Human Interaction, published online on 2026-08-08. Written by Billy Shi and 1 co-authors. 96 references. The full text is freely available (open access, CC BY 4.0).

RezumatDate bibliograficeCiteRelated studies

Abstract (original language)

Applications of large language models (LLMs) in software engineering are soaring quickly. Despite ample evidence supporting LLMs’ capabilities of generating high-quality and creative suggestions, human–LLM interactions in software testing remain under-explored, and there lacks empirical evidence supporting efficacious human–LLM system designs. In software testing, the industry’s pursuit of “autonomous” systems neglects intensive human involvement in prior-design and post-review processes. This article discusses two empirical user studies for a test case brainstorming task: (1) exploring user behaviors in human–LLM interactions compared to web search ( \(N_{1}=16\) ) and (2) investigating three modified interaction strategies—preemptive prompting, buffered response, and guided input ( \(N_{2}=24\) ). We consolidate nine cross-disciplinary metrics to quantitatively evaluate the holistic performance of the human–LLM system, covering three perspectives: test quality, creativity, and attention span. Our findings reveal that users spend 126% more time interacting with LLMs compared to Google search. Moreover, preemptively prompting the LLM system significantly improves test quality and task creativity by over 30%, while simultaneously reducing user idle time by up to 49%. Based on the results, this article discusses three interaction design principles—mixed initiative, acceptability, and appropriation—as guidance for future iterations of an efficacious LLM-assisted software testing system.

Cite

APA

Shi, B., & Kristensson, P. O. (2026). Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development. ACM Transactions on Computer-Human Interaction, 33(4), 1-38. https://doi.org/10.1145/3817601

Vancouver

Shi B, Kristensson PO. Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development. ACM Transactions on Computer-Human Interaction. 2026;33(4):1-38. doi:10.1145/3817601

BibTeX
@article{Shi2026,
  author = {Shi, Billy and Kristensson, Per Ola},
  title = {Preemptive, Buffered, or Guided? Empirical Studies on Human–AI Interaction Strategies for Software Test Case Development},
  journal = {ACM Transactions on Computer-Human Interaction},
  year = {2026},
  volume = {33},
  number = {4},
  pages = {1--38},
  doi = {10.1145/3817601}
}
Start for free →