Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse
Social Science Computer Review · 12.11.2026
DOI: 10.1177/08944393261502769 ↗Kurz gefasst
Fachartikel in der Zeitschrift Social Science Computer Review, online erschienen am 12.11.2026. Verfasst von Yoonhee Shin und 3 weiteren Autor:innen. Die Arbeit nennt 38 Literaturangaben. Gefördert von: Hanyang University.
Abstract (Originalsprache)
Theory-based content analysis is central to social science research, but coding context-dependent interactional discourse remains difficult to scale. This study examines whether fine-tuned large language models (LLMs) can support theory-aligned, episode-level coding of complex social interaction, using socially shared metacognitive regulation (SSMR) in collaborative learning discourse as a demanding empirical case. Using a human-coded dataset of 2,372 discourse episodes from undergraduate collaborative problem-solving, we adapted the large language model content analysis (LACA) framework through an AI-aligned codebook, prompt design, and supervised fine-tuning. Prompt-only and fine-tuned models were compared across episode-level classification and multilayer SSMR coding. Fine-tuned models substantially improved agreement with human coding and predictive performance relative to prompt-only baselines. Performance was strongest for episode-level classification and several higher-support coding layers, whereas errors remained concentrated in sparse, boundary-prone, and highly interpretive categories requiring social-intent reconstruction and temporally unfolding context. The findings extend recent work on LLM-assisted qualitative and content analysis by showing both the promise and limits of supervised, theory-aligned automation for multilayer social interaction coding. Overall, the study supports a human-centered approach to LLM-assisted content analysis, in which automated coding can support more stable coding tasks while human expertise remains essential for ambiguous, boundary-prone, and interpretively demanding judgments.
Förderung
- Hanyang University
Zitieren
APA
Shin, Y., Son, Y., Lee, H., & Kim, M. (2026). Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse. Social Science Computer Review. https://doi.org/10.1177/08944393261502769
Vancouver
Shin Y, Son Y, Lee H, Kim M. Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse. Social Science Computer Review. 2026. doi:10.1177/08944393261502769
BibTeX
@article{Shin2026,
author = {Shin, Yoonhee and Son, Youngkwan and Lee, Haengkyung and Kim, Minyoung},
title = {Fine-Tuned Large Language Models for Theory-Aligned Social Science Content Analysis: Evidence From Collaborative Learning Discourse},
journal = {Social Science Computer Review},
year = {2026},
doi = {10.1177/08944393261502769}
}
EUVIMED