From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews
AERA Open · 2025-07 · 11
DOI: 10.1177/23328584251374595 ↗Kurz gefasst
Fachartikel in der Zeitschrift AERA Open, 2025-10. Verfasst von Alex Liu und 1 weiteren Autor:innen. Die Arbeit wurde bisher 11-mal zitiert (Crossref) und nennt 84 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC 4.0). Gefördert von: William T. Grant Foundation, Ballmer Group, Division of Graduate Education.
Abstract (Originalsprache)
Stakeholder feedback is essential for policymakers to evaluate and develop effective policies, but traditional qualitative analysis methods are often labor-intensive and time-consuming. This study investigates the use of Large Language Models (LLMs) like GPT-4 Turbo (GPT-4) with human expertise to analyze stakeholder interviews regarding K–12 education policy in a U.S. state. The research employed a mixed-methods approach where human experts developed a codebook and iterative prompts for GPT-4 to conduct thematic and sentiment analysis. Results demonstrated that GPT-4’s thematic coding achieved 78% agreement with human coding at detailed levels and 96% alignment for broader themes, exceeding traditional Natural Language Processing methods by over 25%. GPT-4 also produced sentiment analysis results more closely aligned with a human expert’s judgment. Our qualitative comparisons between human and GPT-4 analysis results highlight the complementary roles of human expertise and LLMs in enhancing efficiency, validity, and interpretability of educational policy research.
Förderung
- William T. Grant Foundation
- Ballmer Group
- Division of Graduate Education
Zitieren
APA
Liu, A., & Sun, M. (2025). From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews. AERA Open, 11. https://doi.org/10.1177/23328584251374595
Vancouver
Liu A, Sun M. From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews. AERA Open. 2025;11. doi:10.1177/23328584251374595
BibTeX
@article{Liu2025,
author = {Liu, Alex and Sun, Min},
title = {From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews},
journal = {AERA Open},
year = {2025},
volume = {11},
doi = {10.1177/23328584251374595}
}
EUVIMED