EUVIMEDEUVIMED entdecken
Open Access

Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study

Ziru Fu iD, Yu Cheng Hsu iD, Christian S Chan iD, Chaak Ming Lau iD, Joyce Liu iD, Paul Siu Fai Yip iD

Journal of Medical Internet Research · 30.01.2024 · 26 · e51069

DOI: 10.2196/51069 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Journal of Medical Internet Research, online erschienen am 30.01.2024. Verfasst von Ziru Fu und 5 weiteren Autor:innen. Beteiligte Einrichtung: The Hong Kong Jockey Club Centre for Suicide Research and Prevention, Faculty of Social Sciences, The University of Hong Kong, Hong Kong SAR, China (…. Hauptthemen laut MEDLINE: Asian People, Language, Artificial Intelligence, Communication. Die Arbeit wurde bisher 39-mal zitiert (Crossref) und nennt 28 Literaturangaben. Der Volltext ist frei in PubMed Central verfügbar.

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

Sentiment analysis is a significant yet difficult task in natural language processing. The linguistic peculiarities of Cantonese, including its high similarity with Standard Chinese, its grammatical and lexical uniqueness, and its colloquialism and multilingualism, make it different from other languages and pose additional challenges to sentiment analysis. Recent advances in models such as ChatGPT offer potential viable solutions.

Objective

This study investigated the efficacy of GPT-3.5 and GPT-4 in Cantonese sentiment analysis in the context of web-based counseling and compared their performance with other mainstream methods, including lexicon-based methods and machine learning approaches.

Methods

We analyzed transcripts from a web-based, text-based counseling service in Hong Kong, including a total of 131 individual counseling sessions and 6169 messages between counselors and help-seekers. First, a codebook was developed for human annotation. A simple prompt (“Is the sentiment of this Cantonese text positive, neutral, or negative? Respond with the sentiment label only.”) was then given to GPT-3.5 and GPT-4 to label each message’s sentiment. GPT-3.5 and GPT-4’s performance was compared with a lexicon-based method and 3 state-of-the-art models, including linear regression, support vector machines, and long short-term memory neural networks.

Results

Our findings revealed ChatGPT’s remarkable accuracy in sentiment classification, with GPT-3.5 and GPT-4, respectively, achieving 92.1% (5682/6169) and 95.3% (5880/6169) accuracy in identifying positive, neutral, and negative sentiment, thereby outperforming the traditional lexicon-based method, which had an accuracy of 37.2% (2295/6169), and the 3 machine learning models, which had accuracies ranging from 66% (4072/6169) to 70.9% (4374/6169).

Conclusions

Among many text analysis techniques, ChatGPT demonstrates superior accuracy and emerges as a promising tool for Cantonese sentiment analysis. This study also highlights ChatGPT’s applicability in real-world scenarios, such as monitoring the quality of text-based counseling services and detecting message-level sentiments in vivo. The insights derived from this study pave the way for further exploration into the capabilities of ChatGPT in the context of underresourced languages and specialized domains like psychotherapy and natural language processing.

Schlagwörter

MeSH-Schlagwörter (MEDLINE)

Asian PeopleLanguageArtificial IntelligenceCommunicationHumansCounselorsHong KongLinear Models

Stichwörter der Autor:innen

CantoneseChatGPTNLPcounselingnatural language processingsentiment analysis

Autor:innen und Institutionen

  • The Hong Kong Jockey Club Centre for Suicide Research and Prevention, Faculty of Social Sciences, The University of Hong Kong, Hong Kong SAR, China (Hong Kong).
  • Department of Psychology, The University of Hong Kong, Hong Kong SAR, China (Hong Kong).
  • Department of Psychology and Linguistics, International Christian University, Tokyo, Japan.
  • Department of Linguistics and Modern Language Studies, The Education University of Hong Kong, Hong Kong SAR, China (Hong Kong).
  • Department of Social Work and Social Administration, Faculty of Social Sciences, The University of Hong Kong, Hong Kong SAR, China (Hong Kong).

ORCID-Profile: Ziru Fu, Yu Cheng Hsu, Christian S Chan, Chaak Ming Lau, Joyce Liu, Paul Siu Fai Yip

Zitieren

APA

Fu, Z., Hsu, Y. C., Chan, C. S., Lau, C. M., Liu, J., & Yip, P. S. F. (2024). Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study. Journal of Medical Internet Research, 26, e51069. https://doi.org/10.2196/51069

Vancouver

Fu Z, Hsu YC, Chan CS, Lau CM, Liu J, Yip PSF. Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study. Journal of Medical Internet Research. 2024;26:e51069. doi:10.2196/51069

BibTeX
@article{Fu2024,
  author = {Fu, Ziru and Hsu, Yu Cheng and Chan, Christian S and Lau, Chaak Ming and Liu, Joyce and Yip, Paul Siu Fai},
  title = {Efficacy of ChatGPT in Cantonese Sentiment Analysis: Comparative Study},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  volume = {26},
  pages = {e51069},
  doi = {10.2196/51069}
}
Kostenlos starten →