EUVIMEDScopri EUVIMED
Open access

Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study

Wenxiong Liao iD, Zhengliang Liu iD, Haixing Dai iD, Shaochen Xu iD, Zihao Wu iD, Yiyang Zhang iD, Xiaoke Huang iD, Dajiang Zhu iD, Hongmin Cai iD, Quanzheng Li iD, Tianming Liu iD, Xiang Li iD

JMIR Medical Education · 2023-12-28 · 9 · e48904

DOI: 10.2196/48904 ↗

At a glance

Journal article in JMIR Medical Education, published online on 2023-12-28. Written by Wenxiong Liao and 11 co-authors. Institution: School of Computer Science and Engineering, South China University of Technology, Guangzhou, China. Major MEDLINE topics: Artificial Intelligence, Algorithms. Cited 78 times so far (Crossref); 54 references. The full text is freely available in PubMed Central.

AbstractSubjectsDati bibliograficiCiteRelated studies

Abstract (original language)

Background

Large language models, such as ChatGPT, are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the internet. However, medical texts, such as clinical notes and diagnoses, require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to health care and the general public.

Objective

This study is among the first on responsible artificial intelligence–generated content in medicine. We focus on analyzing the differences between medical texts written by human experts and those generated by ChatGPT and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT.

Methods

We first constructed a suite of data sets containing medical texts written by human experts and generated by ChatGPT. We analyzed the linguistic features of these 2 types of content and uncovered differences in vocabulary, parts-of-speech, dependency, sentiment, perplexity, and other aspects. Finally, we designed and implemented machine learning methods to detect medical text generated by ChatGPT. The data and code used in this paper are published on GitHub.

Results

Medical texts written by humans were more concrete, more diverse, and typically contained more useful information, while medical texts generated by ChatGPT paid more attention to fluency and logic and usually expressed general terminologies rather than effective information specific to the context of the problem. A bidirectional encoder representations from transformers–based model effectively detected medical texts generated by ChatGPT, and the F1 score exceeded 95%.

Conclusions

Although text generated by ChatGPT is grammatically perfect and human-like, the linguistic characteristics of generated medical texts were different from those written by human experts. Medical text generated by ChatGPT could be effectively detected by the proposed machine learning algorithms. This study provides a pathway toward trustworthy and accountable use of large language models in medicine.

Subjects

MeSH terms (MEDLINE)

Artificial IntelligenceAlgorithmsHumansDisinformationElectric Power SuppliesHealth Facilities

Author keywords

ChatGPTartificial intelligencelinguistic analysismachine learningmedical ethicsmedical textstext classification

Authors and institutions

  • School of Computer Science and Engineering, South China University of Technology, Guangzhou, China.
  • School of Computing, University of Georgia, Athens, GA, United States.
  • Department of Computer Science and Engineering, University of Texas at Arlington, Arlington, TX, United States.
  • Department of Radiology, Massachusetts General Hospital, Boston, MA, United States.

ORCID profiles: Wenxiong Liao, Zhengliang Liu, Haixing Dai, Shaochen Xu, Zihao Wu, Yiyang Zhang, Xiaoke Huang, Dajiang Zhu, Hongmin Cai, Quanzheng Li, Tianming Liu, Xiang Li

Cite

APA

Liao, W., Liu, Z., Dai, H., Xu, S., Wu, Z., Zhang, Y., Huang, X., Zhu, D., Cai, H., Li, Q., Liu, T., & Li, X. (2023). Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study. JMIR Medical Education, 9, e48904. https://doi.org/10.2196/48904

Vancouver

Liao W, Liu Z, Dai H, Xu S, Wu Z, Zhang Y, et al. Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study. JMIR Medical Education. 2023;9:e48904. doi:10.2196/48904

BibTeX
@article{Liao2023,
  author = {Liao, Wenxiong and Liu, Zhengliang and Dai, Haixing and Xu, Shaochen and Wu, Zihao and Zhang, Yiyang and Huang, Xiaoke and Zhu, Dajiang and Cai, Hongmin and Li, Quanzheng and Liu, Tianming and Li, Xiang},
  title = {Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study},
  journal = {JMIR Medical Education},
  year = {2023},
  volume = {9},
  pages = {e48904},
  doi = {10.2196/48904}
}
Start for free →