EUVIMEDEUVIMED entdecken
Open Access · CC BY 4.0

The Potential Clinical Utility of the Customized Large Language Model in Gastroenterology: A Pilot Study

Eun Jeong Gong iD, Chang Seok Bang iD, Jae Jun Lee, Jonghyung Park, Eunsil Kim iD, Subeen Kim iD, Minjae Kimm, Seoung-Ho Choi

Bioengineering · 24.12.2024 · 12 (1) · 1

DOI: 10.3390/bioengineering12010001 ↗

Kurz gefasst

Fachartikel in der Zeitschrift Bioengineering, online erschienen am 24.12.2024. Verfasst von Eun Jeong Gong und 7 weiteren Autor:innen. Beteiligte Einrichtung: Department of Internal Medicine, Hallym University College of Medicine, Chuncheon 24253, Republic of Korea. Die Arbeit wurde bisher 14-mal zitiert (Crossref) und nennt 25 Literaturangaben. Der Volltext ist frei zugänglich (Open Access, CC BY 4.0). Gefördert von: Gangwon Information & Culture Industry Promotion Agency (GICA) and National IT Industry Promotion Agency (NIPA), grant funded by the Korea government (MSIT) (D…, Gangwon Information and Culture Industry Promotion Agency (GICA) and the National IT Industry Promotion Agency (NIPA), Korean government (MSIT).

AbstractSchlagwörterBibliografische AngabenZitierenVerwandte Studien

Abstract (Originalsprache)

Background

The large language model (LLM) has the potential to be applied to clinical practice. However, there has been scarce study on this in the field of gastroenterology.

Aim

This study explores the potential clinical utility of two LLMs in the field of gastroenterology: a customized GPT model and a conventional GPT-4o, an advanced LLM capable of retrieval-augmented generation (RAG).

Method

We established a customized GPT with the BM25 algorithm using Open AI’s GPT-4o model, which allows it to produce responses in the context of specific documents including textbooks of internal medicine (in English) and gastroenterology (in Korean). Also, we prepared a conventional ChatGPT 4o (accessed on 16 October 2024) access. The benchmark (written in Korean) consisted of 15 clinical questions developed by four clinical experts, representing typical questions for medical students. The two LLMs, a gastroenterology fellow, and an expert gastroenterologist were tested to assess their performance.

Results

While the customized LLM correctly answered 8 out of 15 questions, the fellow answered 10 correctly. When the standardized Korean medical terms were replaced with English terminology, the LLM’s performance improved, answering two additional knowledge-based questions correctly, matching the fellow’s score. However, judgment-based questions remained a challenge for the model. Even with the implementation of ‘Chain of Thought’ prompt engineering, the customized GPT did not achieve improved reasoning. Conventional GPT-4o achieved the highest score among the AI models (14/15). Although both models performed slightly below the expert gastroenterologist’s level (15/15), they show promising potential for clinical applications (scores comparable with or higher than that of the gastroenterology fellow).

Conclusions

LLMs could be utilized to assist with specialized tasks such as patient counseling. However, RAG capabilities by enabling real-time retrieval of external data not included in the training dataset, appear essential for managing complex, specialized content, and clinician oversight will remain crucial to ensure safe and effective use in clinical practice.

Schlagwörter

Stichwörter der Autor:innen

artificial intelligencegastroenterologylarge language model

Autor:innen und Institutionen

  • Department of Internal Medicine, Hallym University College of Medicine, Chuncheon 24253, Republic of Korea.
  • Institute for Liver and Digestive Diseases, Hallym University, Chuncheon 24253, Republic of Korea.
  • Institute of New Frontier Research, Hallym University College of Medicine, Chuncheon 24253, Republic of Korea.
  • Department of Anesthesiology and Pain Medicine, Hallym University College of Medicine, Chuncheon 24253, Republic of Korea.
  • Meninblox, Inc., Gwangju 61008, Republic of Korea.
  • Department of Plastic Art, Tech University of Korea, Siheung 15073, Republic of Korea.

ORCID-Profile: Eun Jeong Gong, Chang Seok Bang, Eunsil Kim, Subeen Kim

Förderung

  • Gangwon Information & Culture Industry Promotion Agency (GICA) and National IT Industry Promotion Agency (NIPA), grant funded by the Korea government (MSIT) (D…
  • Gangwon Information and Culture Industry Promotion Agency (GICA) and the National IT Industry Promotion Agency (NIPA)
  • Korean government (MSIT)

Zitieren

APA

Gong, E. J., Bang, C. S., Lee, J. J., Park, J., Kim, E., Kim, S., Kimm, M., & Choi, S. H. (2024). The Potential Clinical Utility of the Customized Large Language Model in Gastroenterology: A Pilot Study. Bioengineering, 12(1), 1. https://doi.org/10.3390/bioengineering12010001

Vancouver

Gong EJ, Bang CS, Lee JJ, Park J, Kim E, Kim S, et al. The Potential Clinical Utility of the Customized Large Language Model in Gastroenterology: A Pilot Study. Bioengineering. 2024;12(1):1. doi:10.3390/bioengineering12010001

BibTeX
@article{Gong2024,
  author = {Gong, Eun Jeong and Bang, Chang Seok and Lee, Jae Jun and Park, Jonghyung and Kim, Eunsil and Kim, Subeen and Kimm, Minjae and Choi, Seoung-Ho},
  title = {The Potential Clinical Utility of the Customized Large Language Model in Gastroenterology: A Pilot Study},
  journal = {Bioengineering},
  year = {2024},
  volume = {12},
  number = {1},
  pages = {1},
  doi = {10.3390/bioengineering12010001}
}
Kostenlos starten →