A Systematic Evaluation Framework on Emotion Inference for Large Language Models Using Appraisal Theory
IEEE Transactions on Affective Computing · 2026-07 · 17 (3) · 3626-3639
DOI: 10.1109/taffc.2026.3702035 ↗Kurz gefasst
Fachartikel in der Zeitschrift IEEE Transactions on Affective Computing, 2026-07. Verfasst von Tobias Thejll-Madsen und 2 weiteren Autor:innen. Die Arbeit wurde bisher 1-mal zitiert (Crossref) und nennt 53 Literaturangaben. Gefördert von: UK Research and Innovation.
Abstract (Originalsprache)
We propose and implement a systematic evaluation framework to assess Large Language Models' (LLMs) ability to infer emotions through the lens of Appraisal Theory of Emotion. Our framework generates thousands of story vignettes across six domains ranging from sports to social interactions that can be systematically modified using emotion-relevant features derived from appraisal theory. We first validated these vignettes through human participant responses, which served as validation for the models' responses. Our evaluation included four LLMs: GPT-4o, Claude Sonnet 3.5, Gemini 1.5 Pro, and Llama 3.1 405B. Our results show that all tested LLMs exhibit strong correlations with human responses in five of the six domains, demonstrating sensitivity to emotion-relevant features that align with both theoretical predictions and human ratings. Notably, Claude Sonnet and Llama perform consistently across all domains, while GPT-4o and Gemini Pro diverged from human ratings in one domain, assigning implausible emotional intensities. These inconsistencies suggest that surface-level features can override theoretical manipulations, leading to inconsistent emotion ratings even when models demonstrate robust performance in other domains. Our findings highlight the general alignment between LLMs and human emotional inference when appraisal features are explicitly manipulated, while also demonstrating the necessity of systematic testing to identify domain-specific vulnerabilities. This work provides a framework and recommendations for such systematic evaluations.
Förderung
- UK Research and Innovation
Zitieren
APA
Thejll-Madsen, T., Yongsatianchot, N., & Marsella, S. (2026). A Systematic Evaluation Framework on Emotion Inference for Large Language Models Using Appraisal Theory. IEEE Transactions on Affective Computing, 17(3), 3626-3639. https://doi.org/10.1109/taffc.2026.3702035
Vancouver
Thejll-Madsen T, Yongsatianchot N, Marsella S. A Systematic Evaluation Framework on Emotion Inference for Large Language Models Using Appraisal Theory. IEEE Transactions on Affective Computing. 2026;17(3):3626-3639. doi:10.1109/taffc.2026.3702035
BibTeX
@article{ThejllMadsen2026,
author = {Thejll-Madsen, Tobias and Yongsatianchot, Nutchanon and Marsella, Stacy},
title = {A Systematic Evaluation Framework on Emotion Inference for Large Language Models Using Appraisal Theory},
journal = {IEEE Transactions on Affective Computing},
year = {2026},
volume = {17},
number = {3},
pages = {3626--3639},
doi = {10.1109/taffc.2026.3702035}
}
EUVIMED