Can an experienced qualitative researcher distinguish AI from human qualitative content analysis?
Journal of Healthcare Simulation · 10.08.2026
DOI: 10.54531/fcqk4553 ↗Kurz gefasst
Qualitative Studie in der Zeitschrift Journal of Healthcare Simulation, online erschienen am 10.08.2026. Verfasst von Alexandra T. Lucas und 5 weiteren Autor:innen.
Abstract (Originalsprache)
Background
Artificial intelligence (AI) has become increasingly embedded in research workflows. Large language models (LLMs) are being used to code segments of text, organise codes into themes and interpret patterns within contexts. Recent comparisons between human and AI analyses demonstrate up to 80% thematic overlap, yet humans consistently exhibit deeper interpretive integration and contextual understanding. This study assesses whether experienced researchers can distinguish between entirely human-generated and AI-generated qualitative content analyses of a simulation debriefing.
Methods
We conducted a qualitative descriptive study comparing human-generated qualitative content analysis (QCA) with ChatGPT-4o-generated QCA using a single focus group transcript on emotion management during debriefing of an emotion-triggering simulation case. First, a human research team performed QCA following Schreier’s framework to identify themes. The same transcript was then analysed by ChatGPT-4o using a prompt aligned with Schreier’s QCA approach. Second, both outputs were standardised into matched text formats to control for structural and stylistic variation. Third, de-identified versions of each analysis (human and AI) were provided to a separate group of experienced reviewers, who completed two rounds of open-ended review. Finally, we conducted a QCA of the reviewers’ responses to examine how expert researchers characterise and differentiate AI- and human-generated qualitative analyses.
Results
Only three out of five experienced researchers were able to distinguish between AI-generated and human-generated QCA. The reviewers who correctly identified the AI analysis noted that AI generated incomplete quotes and missed specific details. Reviewers’ preferences for a given analysis did not correlate with accurate identification of human-generated vs AI-generated analysis. Three out of five reviewers preferred the AI analysis.
Discussion
To our knowledge this is the first study specific to simulation debriefing. Two out of five reviewers were not able to differentiate between human and AI-generated outputs, suggesting that confident identification of AI-generated content may reflect bias more than accuracy, and raising concerns about the fairness of informal AI-detection practices. A combined approach in which humans perform an initial analysis that is then followed by subsequent refinement using AI-based processes may represent the most productive path forward.
Zitieren
APA
Lucas, A. T., Ramos, J., Bajwa, M., Calhoun, A., Scerbo, M. W., & Palaganas, J. C. (2026). Can an experienced qualitative researcher distinguish AI from human qualitative content analysis?. Journal of Healthcare Simulation. https://doi.org/10.54531/fcqk4553
Vancouver
Lucas AT, Ramos J, Bajwa M, Calhoun A, Scerbo MW, Palaganas JC. Can an experienced qualitative researcher distinguish AI from human qualitative content analysis?. Journal of Healthcare Simulation. 2026. doi:10.54531/fcqk4553
BibTeX
@article{Lucas2026,
author = {Lucas, Alexandra T. and Ramos, Jianna and Bajwa, Maria and Calhoun, Aaron and Scerbo, Mark W. and Palaganas, Janice C.},
title = {Can an experienced qualitative researcher distinguish AI from human qualitative content analysis?},
journal = {Journal of Healthcare Simulation},
year = {2026},
doi = {10.54531/fcqk4553}
}
EUVIMED