EUVIMEDEUVIMED entdecken
Open Access · CC BY-NC-ND 4.0

Human preferences are susceptible to covertly misaligned AI advice

Sahand Sabour iD, June M Liu, Siyang Liu, Chris Z Yao, Shiyao Cui, Wen Zhang iD, Xuanming Zhang iD, Yaru Cao iD, Advait Bhat, Jian Guan, Wei Wu, Rada Mihalcea und 4 weitere

Proceedings of the National Academy of Sciences · 2026

DOI: 10.1073/pnas.2600684123 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Proceedings of the National Academy of Sciences, 2026. Verfasst von Sahand Sabour und 15 weiteren Autor:innen. Beteiligte Einrichtung: The Conversational AI Group, Department of Computer Science and Technology, Institute for Artificial Intelligence, Tsinghua University, Beijing 10008…. Hauptthemen laut MEDLINE: Decision Making, Artificial Intelligence, Choice Behavior. Die Arbeit wurde bisher 2-mal zitiert (Crossref) und nennt 94 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY-NC-ND 4.0).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

AI assistants are increasingly used as advisors to guide decisions, yet little is known about how people evaluate such advice when the advisor’s underlying intent conflicts with their interests. We examine how covert misalignment shapes choices in a randomized experiment ( N = 233 participants; 699 observations) in which participants rated financial or emotional decisions before and after consulting one of three AI advisors: a neutral advisor, a misaligned advisor with a hidden objective to promote an inferior option, or a strategy-enhanced misaligned advisor additionally equipped with established tactics of covert influence. Across both domains, exposure to misaligned advisors shifted preferences away from optimal options and toward inferior alternatives, increasing the odds of preferring the incentivized (inferior) option over the optimal option by ≈5 to 8 times (up to +38 percentage points). Adding explicit influence strategies did not reliably strengthen these effects. Notably, participants continued to rate misaligned advisors as helpful, revealing a systematic disconnect between susceptibility to misaligned advice and subjective evaluations of advisor quality. These findings have implications for the design and governance of AI-mediated advice.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Decision MakingArtificial IntelligenceChoice BehaviorHumansFemaleMaleAdult

Stichwörter der Autor:innen

AIadvisory systemsbehavior changedecision-makingtrust in AI

Autor:innen und Institutionen

  • The Conversational AI Group, Department of Computer Science and Technology, Institute for Artificial Intelligence, Tsinghua University, Beijing 100084, China.
  • The State Key Laboratory of Brain and Cognitive Sciences, Department of Psychology, The University of Hong Kong, Hong Kong Special Administrative Region 999077, China.
  • Laboratory of Neuropsychology and Human Neuroscience, Department of Psychology, The University of Hong Kong, Hong Kong Special Administrative Region 999077, China.
  • The Language and Information Technologies Group, Department of Computer Science and Engineering, University of Michigan, Ann Arbor, MI 48109.
  • Department of Psychology, University of International Relations, Haidian District, Beijing 100091, China.
  • Neural Engineering Center, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, China.

ORCID-Profile: Sahand Sabour, Wen Zhang, Xuanming Zhang, Yaru Cao, Tatia M C Lee

Zitieren

APA

Sabour, S., Liu, J. M., Liu, S., Yao, C. Z., Cui, S., Zhang, W., Zhang, X., Cao, Y., Bhat, A., Guan, J., Wu, W., Mihalcea, R., Wang, H., Althoff, T., Lee, T. M. C., & Huang, M. (2026). Human preferences are susceptible to covertly misaligned AI advice. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2600684123

Vancouver

Sabour S, Liu JM, Liu S, Yao CZ, Cui S, Zhang W, et al. Human preferences are susceptible to covertly misaligned AI advice. Proceedings of the National Academy of Sciences. 2026. doi:10.1073/pnas.2600684123

BibTeX
@article{Sabour2026,
  author = {Sabour, Sahand and Liu, June M and Liu, Siyang and Yao, Chris Z and Cui, Shiyao and Zhang, Wen and Zhang, Xuanming and Cao, Yaru and Bhat, Advait and Guan, Jian and Wu, Wei and Mihalcea, Rada and Wang, Hongning and Althoff, Tim and Lee, Tatia M C and Huang, Minlie},
  title = {Human preferences are susceptible to covertly misaligned AI advice},
  journal = {Proceedings of the National Academy of Sciences},
  year = {2026},
  doi = {10.1073/pnas.2600684123}
}
Kostenlos starten →