EUVIMEDOpdag EUVIMED
Open access

Capacity of Generative AI to Interpret Human Emotions From Visual and Textual Data: Pilot Evaluation Study

Zohar Elyoseph iD, Elad Refoua iD, Kfir Asraf iD, Maya Lvovsky iD, Yoav Shimoni iD, Dorit Hadar-Shoval iD

JMIR Mental Health · 2024-02-06 · 11 · e54369

DOI: 10.2196/54369 ↗

At a glance

Journal article in JMIR Mental Health, published online on 2024-02-06. Written by Zohar Elyoseph and 5 co-authors. Institution: Department of Educational Psychology, The Center for Psychobiological Research, The Max Stern Yezreel Valley College, Emek Yezreel, Israel. Major MEDLINE topics: Artificial Intelligence, Emotions. Cited 134 times so far (Crossref); 40 references. The full text is freely available in PubMed Central.

AbstractSubjectsBibliografiske oplysningerCiteRelated studies

Abstract (original language)

Background

Mentalization, which is integral to human cognitive processes, pertains to the interpretation of one’s own and others’ mental states, including emotions, beliefs, and intentions. With the advent of artificial intelligence (AI) and the prominence of large language models in mental health applications, questions persist about their aptitude in emotional comprehension. The prior iteration of the large language model from OpenAI, ChatGPT-3.5, demonstrated an advanced capacity to interpret emotions from textual data, surpassing human benchmarks. Given the introduction of ChatGPT-4, with its enhanced visual processing capabilities, and considering Google Bard’s existing visual functionalities, a rigorous assessment of their proficiency in visual mentalizing is warranted.

Objective

The aim of the research was to critically evaluate the capabilities of ChatGPT-4 and Google Bard with regard to their competence in discerning visual mentalizing indicators as contrasted with their textual-based mentalizing abilities.

Methods

The Reading the Mind in the Eyes Test developed by Baron-Cohen and colleagues was used to assess the models’ proficiency in interpreting visual emotional indicators. Simultaneously, the Levels of Emotional Awareness Scale was used to evaluate the large language models’ aptitude in textual mentalizing. Collating data from both tests provided a holistic view of the mentalizing capabilities of ChatGPT-4 and Bard.

Results

ChatGPT-4, displaying a pronounced ability in emotion recognition, secured scores of 26 and 27 in 2 distinct evaluations, significantly deviating from a random response paradigm (P

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceEmotionsHumansPilot ProjectsBenchmarkingEye

Author keywords

AIChatGPTLLMLLMsRMETReading the Mind in the Eyes Testalgorithmalgorithmsartificial intelligenceearly detectionearly warningemotional awarenessemotional comprehensionemotional cueemotional cues

Authors and institutions

  • Department of Educational Psychology, The Center for Psychobiological Research, The Max Stern Yezreel Valley College, Emek Yezreel, Israel.
  • Imperial College London, London, United Kingdom.
  • Department of Psychology, Bar-Ilan University, Ramat Gan, Israel.
  • Department of Psychology, The Max Stern Yezreel Valley College, Emek Yezreel, Israel.
  • Boston Children's Hospital, Boston, MA, United States.

ORCID profiles: Zohar Elyoseph, Elad Refoua, Kfir Asraf, Maya Lvovsky, Yoav Shimoni, Dorit Hadar-Shoval

Cite

APA

Elyoseph, Z., Refoua, E., Asraf, K., Lvovsky, M., Shimoni, Y., & Hadar-Shoval, D. (2024). Capacity of Generative AI to Interpret Human Emotions From Visual and Textual Data: Pilot Evaluation Study. JMIR Mental Health, 11, e54369. https://doi.org/10.2196/54369

Vancouver

Elyoseph Z, Refoua E, Asraf K, Lvovsky M, Shimoni Y, Hadar-Shoval D. Capacity of Generative AI to Interpret Human Emotions From Visual and Textual Data: Pilot Evaluation Study. JMIR Mental Health. 2024;11:e54369. doi:10.2196/54369

BibTeX
@article{Elyoseph2024,
  author = {Elyoseph, Zohar and Refoua, Elad and Asraf, Kfir and Lvovsky, Maya and Shimoni, Yoav and Hadar-Shoval, Dorit},
  title = {Capacity of Generative AI to Interpret Human Emotions From Visual and Textual Data: Pilot Evaluation Study},
  journal = {JMIR Mental Health},
  year = {2024},
  volume = {11},
  pages = {e54369},
  doi = {10.2196/54369}
}
Start for free →