EUVIMEDEUVIMED entdecken
Open Access

Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: Benchmark Study

Prottay Kumar Adhikary iD, Aseem Srivastava iD, Shivani Kumar iD, Salam Michael Singh iD, Puneet Manuja iD, Jini K Gopinath iD, Vijay Krishnan iD, Swati Kedia Gupta iD, Koushik Sinha Deb iD, Tanmoy Chakraborty iD

JMIR Mental Health · 23.07.2024 · 11 · e57306

DOI: 10.2196/57306 ↗

Kurz gefasst

Fachartikel in der Zeitschrift JMIR Mental Health, online erschienen am 23.07.2024. Verfasst von Prottay Kumar Adhikary und 9 weiteren Autor:innen. Beteiligte Einrichtung: Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India. Hauptthemen laut MEDLINE: Benchmarking, Counseling. Die Arbeit wurde bisher 41-mal zitiert (Crossref) und nennt 88 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Comprehensive session summaries enable effective continuity in mental health counseling, facilitating informed therapy planning. However, manual summarization presents a significant challenge, diverting experts’ attention from the core counseling process. Leveraging advances in automatic summarization to streamline the summarization process addresses this issue because this enables mental health professionals to access concise summaries of lengthy therapy sessions, thereby increasing their efficiency. However, existing approaches often overlook the nuanced intricacies inherent in counseling interactions.

Objective

This study evaluates the effectiveness of state-of-the-art large language models (LLMs) in selectively summarizing various components of therapy sessions through aspect-based summarization, aiming to benchmark their performance.

Methods

We first created Mental Health Counseling-Component–Guided Dialogue Summaries, a benchmarking data set that consists of 191 counseling sessions with summaries focused on 3 distinct counseling components (also known as counseling aspects). Next, we assessed the capabilities of 11 state-of-the-art LLMs in addressing the task of counseling-component–guided summarization. The generated summaries were evaluated quantitatively using standard summarization metrics and verified qualitatively by mental health professionals.

Results

Our findings demonstrated the superior performance of task-specific LLMs such as MentalLlama, Mistral, and MentalBART evaluated using standard quantitative metrics such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE)-1, ROUGE-2, ROUGE-L, and Bidirectional Encoder Representations from Transformers Score across all aspects of the counseling components. Furthermore, expert evaluation revealed that Mistral superseded both MentalLlama and MentalBART across 6 parameters: affective attitude, burden, ethicality, coherence, opportunity costs, and perceived effectiveness. However, these models exhibit a common weakness in terms of room for improvement in the opportunity costs and perceived effectiveness metrics.

Conclusions

While LLMs fine-tuned specifically on mental health domain data display better performance based on automatic evaluation scores, expert assessments indicate that these models are not yet reliable for clinical application. Further refinement and validation are necessary before their implementation in practice.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

BenchmarkingCounselingHumansAdultMental DisordersFemale

Stichwörter der Autor:innen

AIartificial intelligencecounseling summarizationdigital healthlarge language modelsmental health

Autor:innen und Institutionen

  • Department of Electrical Engineering, Indian Institute of Technology Delhi, New Delhi, India.
  • Department of Computer Science & Engineering, Indraprastha Institute of Information Technology Delhi, New Delhi, India.
  • YourDOST, Karnataka, India.
  • Department of Psychiatry, All India Institute of Medical Sciences, Rishikesh, India.
  • Department of Psychiatry, All India Institute of Medical Sciences, New Delhi, India.
  • Yardi School of Artificial Intelligence, Indian Institute of Technology Delhi, New Delhi, India.

ORCID-Profile: Prottay Kumar Adhikary, Aseem Srivastava, Shivani Kumar, Salam Michael Singh, Puneet Manuja, Jini K Gopinath, Vijay Krishnan, Swati Kedia Gupta, Koushik Sinha Deb, Tanmoy Chakraborty

Zitieren

APA

Adhikary, P. K., Srivastava, A., Kumar, S., Singh, S. M., Manuja, P., Gopinath, J. K., Krishnan, V., Gupta, S. K., Deb, K. S., & Chakraborty, T. (2024). Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: Benchmark Study. JMIR Mental Health, 11, e57306. https://doi.org/10.2196/57306

Vancouver

Adhikary PK, Srivastava A, Kumar S, Singh SM, Manuja P, Gopinath JK, et al. Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: Benchmark Study. JMIR Mental Health. 2024;11:e57306. doi:10.2196/57306

BibTeX
@article{Adhikary2024,
  author = {Adhikary, Prottay Kumar and Srivastava, Aseem and Kumar, Shivani and Singh, Salam Michael and Manuja, Puneet and Gopinath, Jini K and Krishnan, Vijay and Gupta, Swati Kedia and Deb, Koushik Sinha and Chakraborty, Tanmoy},
  title = {Exploring the Efficacy of Large Language Models in Summarizing Mental Health Counseling Sessions: Benchmark Study},
  journal = {JMIR Mental Health},
  year = {2024},
  volume = {11},
  pages = {e57306},
  doi = {10.2196/57306}
}
Kostenlos starten →