EUVIMEDEUVIMED entdecken
Crossref

Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic

Gustav Verhulsdonck, Jennifer Weible, Danielle Mollie Stambler, Tharon Howard, Jason Tham

Technical Communication · 2024

DOI: 10.55177/tc286621 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Technical Communication, 2024. Verfasst von Gustav Verhulsdonck und 4 weiteren Autor:innen. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 13 Literaturangaben.

AbstractBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Purpose

As technical and professional communicators (TPCs) use AI to develop content, inaccuracies due to AI limitations are introduced; it is vital TPCs evaluate AI-generated content to improve accuracy and human-centeredness. In this article, we present a human-in-the-loop AI content heuristic (HEAT: Human experience, Expertise, Accuracy, and Trust) as a rating mechanism.

Method

This exploratory case study evaluated the quality of content generated by ChatGPT from the perspective of beginner TPC students. We used multiple prompting strategies asking ChatGPT to create documentation on personas using two Darwin Information Type Architecture (DITA) information types namely, concept topics and task instructions, and we evaluated the results with HEAT.

Results

HEAT had good intraclass correlation coefficient (ICC) reliability (.743 pilot; .825 for scenarios) indicating its fitness as a heuristic for evaluating generative AI output. The findings indicate that ChatGPT was generally good at writing concept topics; however, it performed less well creating step-by-step task instructions. Expert TPC input helped develop a better prompt for improved output. We also found that tokenization in ChatGPT (the way it breaks up text) has a large role in terms of noncompliance with format specifications.

Conclusion

There is a need for TPCs to (1) develop new models for AI-assisted content creation, (2) recognize the impact of different prompting strategies on developing specific structured authoring units such as concept and task topics, and (3) be aware of the limitations of AI such as ChatGPT. Human-in-the-loop quality check mechanisms, such as HEAT, can help validate and modify AI-generated content to better serve end users.

Zitieren

APA

Verhulsdonck, G., Weible, J., Stambler, D. M., Howard, T., & Tham, J. (2024). Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic. Technical Communication. https://doi.org/10.55177/tc286621

Vancouver

Verhulsdonck G, Weible J, Stambler DM, Howard T, Tham J. Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic. Technical Communication. 2024. doi:10.55177/tc286621

BibTeX
@article{Verhulsdonck2024,
  author = {Verhulsdonck, Gustav and Weible, Jennifer and Stambler, Danielle Mollie and Howard, Tharon and Tham, Jason},
  title = {Incorporating Human Judgment in AI-Assisted Content Development: The HEAT Heuristic},
  journal = {Technical Communication},
  year = {2024},
  doi = {10.55177/tc286621}
}
Kostenlos starten →