Robustness of large language models in moral judgements
Royal Society Open Science · 2025-04 · 12 (4)
DOI: 10.1098/rsos.241229 ↗Kurz gefasst
Fachartikel in der Zeitschrift Royal Society Open Science, online erschienen am 23.04.2025. Verfasst von Soyoung Oh und 1 weiteren Autor:innen. Beteiligte Einrichtung: Department of Computer Science, Language Science and Technology, Saarland University, Saarbrücken, Germany. Die Arbeit wurde bisher 13-mal zitiert (Crossref) und nennt 63 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: European Research Council.
Abstract (Originalsprache)
With the advent of large language models (LLMs), there has been a growing interest in analysing the preferences encoded in LLMs in the context of morality. Recent work has tested LLMs on various moral judgement tasks and drawn conclusions regarding the alignment between LLMs and humans. The present contribution critically assesses the validity of the method and results employed in previous work for eliciting moral judgements from LLMs. We find that previous results are confounded by biases in the presentation of the options in moral judgement tasks and that LLM responses are highly sensitive to prompt formulation variants as simple as changing ‘Case 1’ and ‘Case 2’ to ‘(A)’ and ‘(B)’. Our results hence indicate that previous conclusions on moral judgements of LLMs cannot be upheld. We make recommendations for more sound methodological setups for future studies.
Schlagwörter
Stichwörter der Autor:innen
Autor:innen und Institutionen
- Department of Computer Science, Language Science and Technology, Saarland University, Saarbrücken, Germany.
ORCID-Profile: Soyoung Oh, Vera Demberg
Förderung
- European Research Council
Zitieren
APA
Oh, S., & Demberg, V. (2025). Robustness of large language models in moral judgements. Royal Society Open Science, 12(4). https://doi.org/10.1098/rsos.241229
Vancouver
Oh S, Demberg V. Robustness of large language models in moral judgements. Royal Society Open Science. 2025;12(4). doi:10.1098/rsos.241229
BibTeX
@article{Oh2025,
author = {Oh, Soyoung and Demberg, Vera},
title = {Robustness of large language models in moral judgements},
journal = {Royal Society Open Science},
year = {2025},
volume = {12},
number = {4},
doi = {10.1098/rsos.241229}
}
EUVIMED