Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic
Technical Communication · 2024
DOI: 10.55177/tc286621 ↗Kurz gefasst
Fachartikel in der Zeitschrift Technical Communication, 2024. Verfasst von Gustav Verhulsdonck und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 13 Literaturangaben.
Abstract (Originalsprache)
Purpose
As technical and professional communicators (TPCs) use AI to develop content, inaccuracies due to AI limitations are introduced; it is vital TPCs evaluate AI-generated content to improve accuracy and human-centeredness. In this article, we present a human-in-the-loop AI content heuristic (HEAT: Human experience, Expertise, Accuracy, and Trust) as a rating mechanism.
Method
This exploratory case study evaluated the quality of content generated by ChatGPT from the perspective of beginner TPC students. We used multiple prompting strategies asking ChatGPT to create documentation on personas using two Darwin Information Type Architecture (DITA) information types namely, concept topics and task instructions, and we evaluated the results with HEAT.
Results
HEAT had good intraclass correlation coefficient (ICC) reliability (.743 pilot; .825 for scenarios) indicating its fitness as a heuristic for evaluating generative AI output. The findings indicate that ChatGPT was generally good at writing concept topics; however, it performed less well creating step-by-step task instructions. Expert TPC input helped develop a better prompt for improved output. We also found that tokenization in ChatGPT (the way it breaks up text) has a large role in terms of noncompliance with format specifications.
Conclusion
There is a need for TPCs to (1) develop new models for AI-assisted content creation, (2) recognize the impact of different prompting strategies on developing specific structured authoring units such as concept and task topics, and (3) be aware of the limitations of AI such as ChatGPT. Human-in-the-loop quality check mechanisms, such as HEAT, can help validate and modify AI-generated content to better serve end users.
Zitieren
APA
Verhulsdonck, G., Weible, J., Stambler, D. M., Howard, T., & Tham, J. (2024). Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic. Technical Communication. https://doi.org/10.55177/tc286621
Vancouver
Verhulsdonck G, Weible J, Stambler DM, Howard T, Tham J. Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic. Technical Communication. 2024. doi:10.55177/tc286621
BibTeX
@article{Verhulsdonck2024,
author = {Verhulsdonck, Gustav and Weible, Jennifer and Stambler, Danielle Mollie and Howard, Tharon and Tham, Jason},
title = {Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic},
journal = {Technical Communication},
year = {2024},
doi = {10.55177/tc286621}
}
EUVIMED